Sep 16 18:42:46.671149 ip-10-0-137-241 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 16 18:42:46.671175 ip-10-0-137-241 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 16 18:42:46.671182 ip-10-0-137-241 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 16 18:42:46.671356 ip-10-0-137-241 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 16 18:42:56.692269 ip-10-0-137-241 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 16 18:42:56.692285 ip-10-0-137-241 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot e009c877bce8432fb06f00568cbd2dea --
Sep 16 18:44:49.812979 ip-10-0-137-241 systemd[1]: Starting Kubernetes Kubelet...
Sep 16 18:44:50.199141 ip-10-0-137-241 kubenswrapper[2554]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:44:50.199141 ip-10-0-137-241 kubenswrapper[2554]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 16 18:44:50.199141 ip-10-0-137-241 kubenswrapper[2554]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:44:50.199141 ip-10-0-137-241 kubenswrapper[2554]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 16 18:44:50.199141 ip-10-0-137-241 kubenswrapper[2554]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 16 18:44:50.201097 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.201021    2554 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 16 18:44:50.205759 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205745    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:50.205759 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205758    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:50.205830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205762    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:50.205830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205766    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:50.205830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205769    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:50.205830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205772    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:50.205830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205775    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:50.205830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205779    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:50.205830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205782    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:50.205830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205785    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:50.205830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205788    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:50.205830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205790    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:50.205830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205793    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:50.205830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205795    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:50.205830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205798    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:50.205830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205801    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:50.205830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205804    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:50.205830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205807    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:50.205830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205809    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:50.205830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205812    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:50.205830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205820    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:50.205830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205823    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:50.206305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205825    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:50.206305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205830    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:50.206305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205834    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:50.206305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205837    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:50.206305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205840    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:50.206305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205843    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:50.206305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205846    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:50.206305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205848    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:50.206305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205851    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:50.206305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205866    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:50.206305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205869    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:50.206305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205872    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:50.206305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205874    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:50.206305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205877    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:50.206305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205881    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:50.206305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205885    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:50.206305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205888    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:50.206305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205890    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:50.206305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205893    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:50.206812 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205896    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:50.206812 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205898    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:50.206812 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205901    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:50.206812 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205904    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:50.206812 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205906    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:50.206812 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205909    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:50.206812 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205911    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:50.206812 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205913    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:50.206812 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205916    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:50.206812 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205918    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:50.206812 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205921    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:50.206812 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205923    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:50.206812 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205926    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:50.206812 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205929    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:50.206812 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205932    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:50.206812 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205935    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:50.206812 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205937    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:50.206812 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205940    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:50.206812 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205944    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:50.206812 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205946    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:50.207316 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205949    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:50.207316 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205951    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:50.207316 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205954    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:50.207316 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205956    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:50.207316 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205960    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:50.207316 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205963    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:50.207316 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205966    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:50.207316 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205969    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:50.207316 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205972    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:50.207316 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205975    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:50.207316 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205978    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:50.207316 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205981    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:50.207316 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205983    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:50.207316 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205986    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:50.207316 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205988    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:50.207316 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205991    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:50.207316 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205994    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:50.207316 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205996    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:50.207316 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.205999    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:50.207843 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206001    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:50.207843 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206003    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:50.207843 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206006    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:50.207843 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206008    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:50.207843 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206011    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:50.207843 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206013    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:50.207843 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206016    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:50.207843 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206020    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:50.207843 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206023    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:50.207843 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206026    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:50.207843 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206028    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:50.207843 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206031    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:50.207843 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206034    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:50.207843 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206037    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:50.207843 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206039    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:50.207843 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206042    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:50.207843 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206044    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:50.207843 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206047    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:50.207843 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206049    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:50.208305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206728    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:50.208305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206734    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:50.208305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206738    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:50.208305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206740    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:50.208305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206743    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:50.208305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206746    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:50.208305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206749    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:50.208305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206752    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:50.208305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206754    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:50.208305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206757    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:50.208305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206759    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:50.208305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206762    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:50.208305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206765    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:50.208305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206767    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:50.208305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206770    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:50.208305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206774    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:50.208305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206778    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:50.208305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206781    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:50.208305 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206784    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:50.208794 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206786    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:50.208794 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206790    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:50.208794 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206792    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:50.208794 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206795    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:50.208794 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206798    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:50.208794 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206800    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:50.208794 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206803    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:50.208794 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206806    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:50.208794 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206808    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:50.208794 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206811    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:50.208794 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206814    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:50.208794 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206816    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:50.208794 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206819    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:50.208794 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206822    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:50.208794 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206824    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:50.208794 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206827    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:50.208794 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206829    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:50.208794 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206832    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:50.208794 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206834    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:50.208794 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206837    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:50.209287 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206840    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:50.209287 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206842    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:50.209287 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206845    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:50.209287 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206848    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:50.209287 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206850    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:50.209287 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206853    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:50.209287 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206855    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:50.209287 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206858    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:50.209287 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206860    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:50.209287 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206863    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:50.209287 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206865    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:50.209287 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206868    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:50.209287 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206870    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:50.209287 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206874    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:50.209287 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206876    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:50.209287 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206879    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:50.209287 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206882    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:50.209287 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206884    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:50.209287 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206887    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:50.209287 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206890    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:50.209779 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206893    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:50.209779 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206895    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:50.209779 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206899    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:50.209779 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206903    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:50.209779 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206906    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:50.209779 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206909    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:50.209779 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206912    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:50.209779 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206915    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:50.209779 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206918    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:50.209779 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206920    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:50.209779 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206923    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:50.209779 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206925    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:50.209779 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206930    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:50.209779 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206933    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:50.209779 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206936    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:50.209779 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206938    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:50.209779 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206941    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:50.209779 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206944    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:50.209779 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206946    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:50.210239 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206949    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:50.210239 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206951    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:50.210239 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206954    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:50.210239 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206956    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:50.210239 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206959    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:50.210239 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206962    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:50.210239 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206966    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:50.210239 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206969    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:50.210239 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206972    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:50.210239 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206975    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:50.210239 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206977    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:50.210239 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206980    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:50.210239 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206982    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:50.210239 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206985    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:50.210239 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206988    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:50.210239 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206991    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:50.210239 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206993    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:50.210239 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206996    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:50.210239 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.206998    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:50.210239 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207001    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:50.210745 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207004    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:50.210745 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207377    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:50.210745 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207382    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:50.210745 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207385    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:50.210745 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207389    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:50.210745 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207394    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:50.210745 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207397    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:50.210745 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207402    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:50.210745 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207405    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:50.210745 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207407    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:50.210745 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207410    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:50.210745 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207413    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:50.210745 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207416    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:50.210745 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207418    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:50.210745 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207421    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:50.210745 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207423    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:50.210745 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207426    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:50.210745 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207429    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:50.210745 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207431    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:50.211326 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207434    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:50.211326 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207437    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:50.211326 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207440    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:50.211326 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207443    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:50.211326 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207445    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:50.211326 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207448    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:50.211326 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207451    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:50.211326 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207453    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:50.211326 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207456    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:50.211326 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207458    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:50.211326 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207461    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:50.211326 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207463    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:50.211326 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207466    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:50.211326 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207468    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:50.211326 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207471    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:50.211326 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207473    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:50.211326 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207476    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:50.211326 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207478    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:50.211326 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207480    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:50.211326 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207483    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:50.211830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207487    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:50.211830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207490    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:50.211830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207492    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:50.211830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207495    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:50.211830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207497    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:50.211830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207500    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:50.211830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207502    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:50.211830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207505    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:50.211830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207507    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:50.211830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207510    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:50.211830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207512    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:50.211830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207515    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:50.211830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207518    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:50.211830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207520    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:50.211830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207523    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:50.211830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207525    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:50.211830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207528    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:50.211830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207531    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:50.211830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207533    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:50.211830 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207535    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:50.212355 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207538    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:50.212355 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207540    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:50.212355 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207543    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:50.212355 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207545    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:50.212355 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207547    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:50.212355 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207550    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:50.212355 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207552    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:50.212355 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207555    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:50.212355 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207558    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:50.212355 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207560    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:50.212355 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207563    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:50.212355 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207565    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:50.212355 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207569    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:50.212355 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207573    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:50.212355 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207577    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:50.212355 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207580    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:50.212355 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207583    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:50.212355 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207586    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:50.212355 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207588    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:50.212355 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207591    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:50.212890 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207594    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:50.212890 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207596    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:50.212890 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207598    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:50.212890 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207601    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:50.212890 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207605    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:50.212890 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207607    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:50.212890 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207610    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:50.212890 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207612    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:50.212890 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207615    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:50.212890 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207618    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:50.212890 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207620    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:50.212890 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207623    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:50.212890 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207626    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:50.212890 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207629    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:50.212890 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207631    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:50.212890 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207634    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:50.212890 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207636    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:50.212890 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207639    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:50.212890 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207641    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:50.212890 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207645    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:50.213369 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.207648    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:50.213369 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207713    2554 flags.go:64] FLAG: --address="0.0.0.0"
Sep 16 18:44:50.213369 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207770    2554 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 16 18:44:50.213369 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207782    2554 flags.go:64] FLAG: --anonymous-auth="true"
Sep 16 18:44:50.213369 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207792    2554 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 16 18:44:50.213369 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207797    2554 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 16 18:44:50.213369 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207802    2554 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 16 18:44:50.213369 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207808    2554 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 16 18:44:50.213369 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207812    2554 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 16 18:44:50.213369 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207817    2554 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 16 18:44:50.213369 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207821    2554 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 16 18:44:50.213369 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207825    2554 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 16 18:44:50.213369 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207828    2554 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 16 18:44:50.213369 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207831    2554 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 16 18:44:50.213369 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207834    2554 flags.go:64] FLAG: --cgroup-root=""
Sep 16 18:44:50.213369 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207836    2554 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 16 18:44:50.213369 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207839    2554 flags.go:64] FLAG: --client-ca-file=""
Sep 16 18:44:50.213369 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207842    2554 flags.go:64] FLAG: --cloud-provider="external"
Sep 16 18:44:50.213369 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207845    2554 flags.go:64] FLAG: --cluster-dns="[]"
Sep 16 18:44:50.213369 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207850    2554 flags.go:64] FLAG: --cluster-domain=""
Sep 16 18:44:50.213369 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207853    2554 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 16 18:44:50.213369 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207856    2554 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 16 18:44:50.213369 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207859    2554 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 16 18:44:50.213369 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207862    2554 flags.go:64] FLAG: --container-log-max-files="5"
Sep 16 18:44:50.213989 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207866    2554 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 16 18:44:50.213989 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207869    2554 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 16 18:44:50.213989 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207872    2554 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 16 18:44:50.213989 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207875    2554 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 16 18:44:50.213989 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207878    2554 flags.go:64] FLAG: --contention-profiling="false"
Sep 16 18:44:50.213989 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207881    2554 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 16 18:44:50.213989 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207884    2554 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 16 18:44:50.213989 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207887    2554 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 16 18:44:50.213989 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207891    2554 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 16 18:44:50.213989 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207895    2554 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 16 18:44:50.213989 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207898    2554 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 16 18:44:50.213989 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207901    2554 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 16 18:44:50.213989 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207905    2554 flags.go:64] FLAG: --enable-load-reader="false"
Sep 16 18:44:50.213989 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207907    2554 flags.go:64] FLAG: --enable-server="true"
Sep 16 18:44:50.213989 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207912    2554 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 16 18:44:50.213989 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207916    2554 flags.go:64] FLAG: --event-burst="100"
Sep 16 18:44:50.213989 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207919    2554 flags.go:64] FLAG: --event-qps="50"
Sep 16 18:44:50.213989 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207922    2554 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 16 18:44:50.213989 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207925    2554 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 16 18:44:50.213989 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207928    2554 flags.go:64] FLAG: --eviction-hard=""
Sep 16 18:44:50.213989 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207931    2554 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 16 18:44:50.213989 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207934    2554 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 16 18:44:50.213989 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207937    2554 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 16 18:44:50.213989 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207940    2554 flags.go:64] FLAG: --eviction-soft=""
Sep 16 18:44:50.213989 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207943    2554 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 16 18:44:50.214597 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207946    2554 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 16 18:44:50.214597 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207948    2554 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 16 18:44:50.214597 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207951    2554 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 16 18:44:50.214597 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207954    2554 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 16 18:44:50.214597 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207957    2554 flags.go:64] FLAG: --fail-swap-on="true"
Sep 16 18:44:50.214597 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207960    2554 flags.go:64] FLAG: --feature-gates=""
Sep 16 18:44:50.214597 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207968    2554 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 16 18:44:50.214597 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207971    2554 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 16 18:44:50.214597 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207974    2554 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 16 18:44:50.214597 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207977    2554 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 16 18:44:50.214597 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207980    2554 flags.go:64] FLAG: --healthz-port="10248"
Sep 16 18:44:50.214597 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207983    2554 flags.go:64] FLAG: --help="false"
Sep 16 18:44:50.214597 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207986    2554 flags.go:64] FLAG: --hostname-override="ip-10-0-137-241.ec2.internal"
Sep 16 18:44:50.214597 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207989    2554 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 16 18:44:50.214597 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207992    2554 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 16 18:44:50.214597 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.207997    2554 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 16 18:44:50.214597 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208000    2554 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 16 18:44:50.214597 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208004    2554 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 16 18:44:50.214597 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208007    2554 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 16 18:44:50.214597 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208011    2554 flags.go:64] FLAG: --image-service-endpoint=""
Sep 16 18:44:50.214597 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208014    2554 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 16 18:44:50.214597 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208017    2554 flags.go:64] FLAG: --kube-api-burst="100"
Sep 16 18:44:50.214597 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208020    2554 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 16 18:44:50.215213 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208023    2554 flags.go:64] FLAG: --kube-api-qps="50"
Sep 16 18:44:50.215213 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208026    2554 flags.go:64] FLAG: --kube-reserved=""
Sep 16 18:44:50.215213 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208029    2554 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 16 18:44:50.215213 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208032    2554 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 16 18:44:50.215213 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208035    2554 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 16 18:44:50.215213 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208037    2554 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 16 18:44:50.215213 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208040    2554 flags.go:64] FLAG: --lock-file=""
Sep 16 18:44:50.215213 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208043    2554 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 16 18:44:50.215213 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208046    2554 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 16 18:44:50.215213 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208049    2554 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 16 18:44:50.215213 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208054    2554 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 16 18:44:50.215213 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208057    2554 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 16 18:44:50.215213 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208060    2554 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 16 18:44:50.215213 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208063    2554 flags.go:64] FLAG: --logging-format="text"
Sep 16 18:44:50.215213 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208066    2554 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 16 18:44:50.215213 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208069    2554 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 16 18:44:50.215213 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208072    2554 flags.go:64] FLAG: --manifest-url=""
Sep 16 18:44:50.215213 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208075    2554 flags.go:64] FLAG: --manifest-url-header=""
Sep 16 18:44:50.215213 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208079    2554 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 16 18:44:50.215213 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208082    2554 flags.go:64] FLAG: --max-open-files="1000000"
Sep 16 18:44:50.215213 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208086    2554 flags.go:64] FLAG: --max-pods="110"
Sep 16 18:44:50.215213 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208089    2554 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 16 18:44:50.215213 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208092    2554 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 16 18:44:50.215213 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208094    2554 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 16 18:44:50.215213 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208099    2554 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 16 18:44:50.215852 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208102    2554 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 16 18:44:50.215852 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208105    2554 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 16 18:44:50.215852 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208108    2554 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 16 18:44:50.215852 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208116    2554 flags.go:64] FLAG: --node-status-max-images="50"
Sep 16 18:44:50.215852 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208119    2554 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 16 18:44:50.215852 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208122    2554 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 16 18:44:50.215852 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208125    2554 flags.go:64] FLAG: --pod-cidr=""
Sep 16 18:44:50.215852 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208129    2554 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 16 18:44:50.215852 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208134    2554 flags.go:64] FLAG: --pod-manifest-path=""
Sep 16 18:44:50.215852 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208137    2554 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 16 18:44:50.215852 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208140    2554 flags.go:64] FLAG: --pods-per-core="0"
Sep 16 18:44:50.215852 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208143    2554 flags.go:64] FLAG: --port="10250"
Sep 16 18:44:50.215852 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208146    2554 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 16 18:44:50.215852 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208149    2554 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07e598cef8b823f41"
Sep 16 18:44:50.215852 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208152    2554 flags.go:64] FLAG: --qos-reserved=""
Sep 16 18:44:50.215852 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208155    2554 flags.go:64] FLAG: --read-only-port="10255"
Sep 16 18:44:50.215852 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208158    2554 flags.go:64] FLAG: --register-node="true"
Sep 16 18:44:50.215852 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208161    2554 flags.go:64] FLAG: --register-with-taints=""
Sep 16 18:44:50.215852 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208164    2554 flags.go:64] FLAG: --registry-burst="10"
Sep 16 18:44:50.215852 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208167    2554 flags.go:64] FLAG: --registry-qps="5"
Sep 16 18:44:50.215852 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208169    2554 flags.go:64] FLAG: --reserved-cpus=""
Sep 16 18:44:50.215852 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208172    2554 flags.go:64] FLAG: --reserved-memory=""
Sep 16 18:44:50.215852 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208176    2554 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 16 18:44:50.215852 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208179    2554 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 16 18:44:50.215852 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208182    2554 flags.go:64] FLAG: --rotate-certificates="false"
Sep 16 18:44:50.216466 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208185    2554 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 16 18:44:50.216466 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208188    2554 flags.go:64] FLAG: --runonce="false"
Sep 16 18:44:50.216466 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208191    2554 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 16 18:44:50.216466 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208194    2554 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 16 18:44:50.216466 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208196    2554 flags.go:64] FLAG: --seccomp-default="false"
Sep 16 18:44:50.216466 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208199    2554 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 16 18:44:50.216466 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208202    2554 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 16 18:44:50.216466 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208207    2554 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 16 18:44:50.216466 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208210    2554 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 16 18:44:50.216466 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208213    2554 flags.go:64] FLAG: --storage-driver-password="root"
Sep 16 18:44:50.216466 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208217    2554 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 16 18:44:50.216466 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208220    2554 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 16 18:44:50.216466 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208223    2554 flags.go:64] FLAG: --storage-driver-user="root"
Sep 16 18:44:50.216466 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208225    2554 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 16 18:44:50.216466 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208229    2554 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 16 18:44:50.216466 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208232    2554 flags.go:64] FLAG: --system-cgroups=""
Sep 16 18:44:50.216466 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208235    2554 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 16 18:44:50.216466 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208240    2554 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 16 18:44:50.216466 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208244    2554 flags.go:64] FLAG: --tls-cert-file=""
Sep 16 18:44:50.216466 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208246    2554 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 16 18:44:50.216466 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208250    2554 flags.go:64] FLAG: --tls-min-version=""
Sep 16 18:44:50.216466 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208252    2554 flags.go:64] FLAG: --tls-private-key-file=""
Sep 16 18:44:50.216466 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208255    2554 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 16 18:44:50.216466 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208258    2554 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 16 18:44:50.216466 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208261    2554 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 16 18:44:50.217086 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208264    2554 flags.go:64] FLAG: --v="2"
Sep 16 18:44:50.217086 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208268    2554 flags.go:64] FLAG: --version="false"
Sep 16 18:44:50.217086 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208272    2554 flags.go:64] FLAG: --vmodule=""
Sep 16 18:44:50.217086 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208275    2554 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 16 18:44:50.217086 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.208279    2554 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 16 18:44:50.217086 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208387    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:50.217086 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208391    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:50.217086 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208394    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:50.217086 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208397    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:50.217086 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208401    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:50.217086 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208405    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:50.217086 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208408    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:50.217086 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208411    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:50.217086 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208414    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:50.217086 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208418    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:50.217086 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208421    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:50.217086 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208424    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:50.217086 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208428    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:50.217086 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208431    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:50.217086 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208434    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:50.217614 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208438    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:50.217614 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208442    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:50.217614 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208453    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:50.217614 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208457    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:50.217614 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208460    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:50.217614 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208463    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:50.217614 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208466    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:50.217614 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208468    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:50.217614 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208471    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:50.217614 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208474    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:50.217614 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208477    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:50.217614 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208480    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:50.217614 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208482    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:50.217614 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208485    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:50.217614 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208488    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:50.217614 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208490    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:50.217614 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208493    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:50.217614 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208495    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:50.217614 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208498    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:50.218095 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208501    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:50.218095 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208504    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:50.218095 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208506    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:50.218095 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208509    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:50.218095 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208511    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:50.218095 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208514    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:50.218095 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208517    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:50.218095 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208521    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:50.218095 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208523    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:50.218095 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208526    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:50.218095 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208530    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:50.218095 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208533    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:50.218095 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208535    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:50.218095 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208539    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:50.218095 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208543    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:50.218095 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208546    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:50.218095 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208549    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:50.218095 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208552    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:50.218095 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208555    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:50.218095 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208557    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:50.218787 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208560    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:50.218787 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208563    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:50.218787 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208565    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:50.218787 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208568    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:50.218787 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208570    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:50.218787 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208573    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:50.218787 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208576    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:50.218787 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208578    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:50.218787 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208581    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:50.218787 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208583    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:50.218787 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208586    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:50.218787 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208588    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:50.218787 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208591    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:50.218787 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208593    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:50.218787 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208596    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:50.218787 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208599    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:50.218787 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208601    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:50.218787 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208604    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:50.218787 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208606    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:50.218787 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208611    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:50.219529 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208614    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:50.219529 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208616    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:50.219529 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208620    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:50.219529 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208624    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:50.219529 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208626    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:50.219529 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208629    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:50.219529 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208631    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:50.219529 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208634    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:50.219529 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208636    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:50.219529 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208639    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:50.219529 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208642    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:50.219529 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208644    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:50.219529 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208647    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:50.219529 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208650    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:50.219529 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208652    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:50.219529 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208655    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:50.219529 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208658    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:50.219529 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208661    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:50.219529 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208664    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:50.219529 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208666    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:50.220051 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208669    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:50.220051 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208672    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:50.220051 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208675    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:50.220051 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208677    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:50.220051 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.208680    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:50.220051 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.209484    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:44:50.220051 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.216364    2554 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 16 18:44:50.220051 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.216381    2554 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 16 18:44:50.220051 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.216398    2554 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 16 18:44:50.220051 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.216404    2554 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 16 18:44:50.220051 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216466    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:50.220051 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216470    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:50.220051 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216474    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:50.220051 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216477    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:50.220051 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216481    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:50.220427 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216484    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:50.220427 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216487    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:50.220427 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216490    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:50.220427 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216493    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:50.220427 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216496    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:50.220427 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216498    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:50.220427 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216501    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:50.220427 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216504    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:50.220427 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216507    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:50.220427 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216509    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:50.220427 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216512    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:50.220427 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216514    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:50.220427 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216517    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:50.220427 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216519    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:50.220427 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216522    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:50.220427 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216525    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:50.220427 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216527    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:50.220427 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216529    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:50.220427 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216532    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:50.220427 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216534    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:50.220934 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216537    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:50.220934 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216539    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:50.220934 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216542    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:50.220934 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216545    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:50.220934 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216554    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:50.220934 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216557    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:50.220934 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216560    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:50.220934 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216563    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:50.220934 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216567    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:50.220934 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216571    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:50.220934 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216573    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:50.220934 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216576    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:50.220934 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216579    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:50.220934 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216583    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:50.220934 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216586    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:50.220934 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216589    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:50.220934 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216592    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:50.220934 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216595    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:50.220934 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216597    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:50.221441 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216600    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:50.221441 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216603    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:50.221441 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216605    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:50.221441 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216608    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:50.221441 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216610    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:50.221441 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216613    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:50.221441 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216615    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:50.221441 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216618    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:50.221441 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216621    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:50.221441 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216623    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:50.221441 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216626    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:50.221441 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216628    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:50.221441 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216632    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:50.221441 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216635    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:50.221441 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216638    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:50.221441 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216640    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:50.221441 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216643    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:50.221441 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216647    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:50.221441 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216655    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:50.221441 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216658    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:50.221951 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216660    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:50.221951 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216663    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:50.221951 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216665    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:50.221951 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216668    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:50.221951 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216671    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:50.221951 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216673    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:50.221951 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216676    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:50.221951 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216678    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:50.221951 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216681    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:50.221951 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216684    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:50.221951 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216686    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:50.221951 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216689    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:50.221951 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216691    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:50.221951 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216694    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:50.221951 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216696    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:50.221951 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216699    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:50.221951 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216702    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:50.221951 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216704    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:50.221951 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216707    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:50.221951 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216710    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:50.222429 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216712    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:50.222429 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216730    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:50.222429 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216733    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:50.222429 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216735    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:50.222429 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216738    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:50.222429 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216740    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:50.222429 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216743    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:50.222429 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216746    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:50.222429 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216748    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:50.222429 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216751    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:50.222429 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216754    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:50.222429 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216762    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:50.222429 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216765    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:50.222429 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216768    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:50.222429 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216770    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:50.222803 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.216776    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:44:50.222803 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216916    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 16 18:44:50.222803 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216921    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 16 18:44:50.222803 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216925    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 16 18:44:50.222803 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216928    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 16 18:44:50.222803 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216931    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 16 18:44:50.222803 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216934    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 16 18:44:50.222803 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216937    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 16 18:44:50.222803 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216940    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 16 18:44:50.222803 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216943    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 16 18:44:50.222803 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216945    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 16 18:44:50.222803 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216948    2554 feature_gate.go:328] unrecognized feature gate: Example
Sep 16 18:44:50.222803 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216951    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 16 18:44:50.222803 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216954    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 16 18:44:50.222803 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216956    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 16 18:44:50.223171 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216959    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 16 18:44:50.223171 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216961    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 16 18:44:50.223171 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216964    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 16 18:44:50.223171 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216966    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 16 18:44:50.223171 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216969    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 16 18:44:50.223171 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216972    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 16 18:44:50.223171 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216974    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 16 18:44:50.223171 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216977    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 16 18:44:50.223171 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216979    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 16 18:44:50.223171 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216982    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 16 18:44:50.223171 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216984    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 16 18:44:50.223171 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216987    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 16 18:44:50.223171 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216989    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 16 18:44:50.223171 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.216993    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 16 18:44:50.223171 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217003    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 16 18:44:50.223171 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217007    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 16 18:44:50.223171 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217010    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 16 18:44:50.223171 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217013    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 16 18:44:50.223171 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217016    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 16 18:44:50.223638 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217019    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 16 18:44:50.223638 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217022    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 16 18:44:50.223638 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217025    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 16 18:44:50.223638 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217028    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 16 18:44:50.223638 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217031    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 16 18:44:50.223638 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217034    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 16 18:44:50.223638 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217037    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 16 18:44:50.223638 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217039    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 16 18:44:50.223638 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217042    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 16 18:44:50.223638 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217044    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 16 18:44:50.223638 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217047    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 16 18:44:50.223638 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217049    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 16 18:44:50.223638 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217052    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 16 18:44:50.223638 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217054    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 16 18:44:50.223638 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217057    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 16 18:44:50.223638 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217059    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 16 18:44:50.223638 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217062    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 16 18:44:50.223638 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217064    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 16 18:44:50.223638 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217067    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 16 18:44:50.223638 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217069    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 16 18:44:50.224141 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217071    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 16 18:44:50.224141 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217074    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 16 18:44:50.224141 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217076    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 16 18:44:50.224141 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217079    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 16 18:44:50.224141 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217082    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 16 18:44:50.224141 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217084    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 16 18:44:50.224141 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217087    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 16 18:44:50.224141 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217090    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 16 18:44:50.224141 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217099    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 16 18:44:50.224141 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217102    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 16 18:44:50.224141 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217104    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 16 18:44:50.224141 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217107    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 16 18:44:50.224141 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217110    2554 feature_gate.go:328] unrecognized feature gate: Example2
Sep 16 18:44:50.224141 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217112    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 16 18:44:50.224141 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217115    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 16 18:44:50.224141 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217117    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 16 18:44:50.224141 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217120    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 16 18:44:50.224141 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217123    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 16 18:44:50.224141 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217125    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 16 18:44:50.224141 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217128    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 16 18:44:50.224622 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217130    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 16 18:44:50.224622 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217132    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 16 18:44:50.224622 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217135    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 16 18:44:50.224622 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217137    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 16 18:44:50.224622 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217140    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 16 18:44:50.224622 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217142    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 16 18:44:50.224622 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217145    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 16 18:44:50.224622 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217147    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 16 18:44:50.224622 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217150    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 16 18:44:50.224622 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217152    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 16 18:44:50.224622 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217155    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 16 18:44:50.224622 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217157    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 16 18:44:50.224622 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217160    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 16 18:44:50.224622 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217163    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 16 18:44:50.224622 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217165    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 16 18:44:50.224622 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217168    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 16 18:44:50.224622 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217170    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 16 18:44:50.224622 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217173    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 16 18:44:50.224622 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217176    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 16 18:44:50.224622 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217178    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 16 18:44:50.225238 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217181    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 16 18:44:50.225238 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217193    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 16 18:44:50.225238 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217201    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 16 18:44:50.225238 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217203    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 16 18:44:50.225238 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217206    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 16 18:44:50.225238 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:50.217208    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 16 18:44:50.225238 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.217213    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 16 18:44:50.225238 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.217323    2554 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 16 18:44:50.225238 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.222349    2554 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 16 18:44:50.225238 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.223269    2554 server.go:1019] "Starting client certificate rotation"
Sep 16 18:44:50.225238 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.223363    2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:44:50.225238 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.223400    2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 16 18:44:50.225697 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.225685    2554 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:44:50.226764 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.226737    2554 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 16 18:44:50.237648 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.237634    2554 log.go:25] "Validated CRI v1 runtime API"
Sep 16 18:44:50.241663 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.241649    2554 log.go:25] "Validated CRI v1 image API"
Sep 16 18:44:50.243097 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.243080    2554 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 16 18:44:50.248389 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.248371    2554 fs.go:135] Filesystem UUIDs: map[1511d844-d036-4e94-8b7a-0b8c6728fe6a:/dev/nvme0n1p4 5bc955e4-f5a1-4226-a2f4-8a966fc69d21:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 16 18:44:50.248450 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.248389    2554 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 16 18:44:50.254048 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.254033    2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:44:50.254171 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.254055    2554 manager.go:216] Machine: {Timestamp:2026-09-16 18:44:50.25269108 +0000 UTC m=+0.328460704 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3103469 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24e4c2d7523b503d2816df4f65dff8 SystemUUID:ec24e4c2-d752-3b50-3d28-16df4f65dff8 BootID:e009c877-bce8-432f-b06f-00568cbd2dea Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e0:20:bd:6b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e0:20:bd:6b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:5b:e5:a9:10:96 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 16 18:44:50.254171 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.254168    2554 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 16 18:44:50.254279 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.254267    2554 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 16 18:44:50.256887 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.256864    2554 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 16 18:44:50.257008 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.256890    2554 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-241.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 16 18:44:50.257606 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.257597    2554 topology_manager.go:138] "Creating topology manager with none policy"
Sep 16 18:44:50.257635 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.257609    2554 container_manager_linux.go:309] "Creating device plugin manager"
Sep 16 18:44:50.257635 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.257616    2554 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:44:50.258308 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.258298    2554 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 16 18:44:50.259023 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.259013    2554 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 16 18:44:50.262953 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.262942    2554 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:44:50.263053 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.263044    2554 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 16 18:44:50.263084 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.263071    2554 kubelet.go:486] "Attempting to sync node with API server"
Sep 16 18:44:50.263084 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.263082    2554 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 16 18:44:50.263143 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.263095    2554 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 16 18:44:50.263143 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.263103    2554 kubelet.go:398] "Adding apiserver pod source"
Sep 16 18:44:50.263143 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.263120    2554 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 16 18:44:50.263518 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.263498    2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:44:50.263518 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.263517    2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 16 18:44:50.267772 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.267753    2554 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 16 18:44:50.269654 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.269639    2554 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 16 18:44:50.269710 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.269668    2554 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 16 18:44:50.270944 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.270927    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 16 18:44:50.270944 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.270944    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 16 18:44:50.271050 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.270950    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 16 18:44:50.271050 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.270956    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 16 18:44:50.271050 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.270964    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 16 18:44:50.271050 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.270970    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 16 18:44:50.271050 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.270983    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 16 18:44:50.271050 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.270988    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 16 18:44:50.271050 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.270994    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 16 18:44:50.271050 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.271000    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 16 18:44:50.271050 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.271008    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 16 18:44:50.271050 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.271014    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 16 18:44:50.271622 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.271612    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 16 18:44:50.271652 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.271623    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 16 18:44:50.275329 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.275314    2554 server.go:1268] "Started kubelet"
Sep 16 18:44:50.275400 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:50.275353    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-241.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 16 18:44:50.275400 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:50.275378    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 16 18:44:50.275400 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.275391    2554 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-241.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 16 18:44:50.275549 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.275404    2554 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 16 18:44:50.275549 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.275443    2554 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 16 18:44:50.275630 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.275551    2554 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 16 18:44:50.276092 ip-10-0-137-241 systemd[1]: Started Kubernetes Kubelet.
Sep 16 18:44:50.277372 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.277356    2554 server.go:310] "Adding debug handlers to kubelet server"
Sep 16 18:44:50.277627 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.277601    2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-srr5x"
Sep 16 18:44:50.279164 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.279051    2554 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 16 18:44:50.281011 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:50.280991    2554 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 16 18:44:50.282656 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:50.281802    2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-241.ec2.internal.18d5e16e2d7a5dcf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-241.ec2.internal,UID:ip-10-0-137-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-241.ec2.internal,},FirstTimestamp:2026-09-16 18:44:50.275286479 +0000 UTC m=+0.351056101,LastTimestamp:2026-09-16 18:44:50.275286479 +0000 UTC m=+0.351056101,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-241.ec2.internal,}"
Sep 16 18:44:50.282656 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.282643    2554 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 16 18:44:50.282788 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.282659    2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 16 18:44:50.282909 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:50.282870    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-241.ec2.internal\" not found"
Sep 16 18:44:50.283290 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.283230    2554 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 16 18:44:50.283290 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.283228    2554 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 16 18:44:50.283290 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.283253    2554 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 16 18:44:50.283481 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.283346    2554 reconstruct.go:97] "Volume reconstruction finished"
Sep 16 18:44:50.283481 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.283355    2554 reconciler.go:29] "Reconciler: start to sync state"
Sep 16 18:44:50.283575 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.283491    2554 factory.go:55] Registering systemd factory
Sep 16 18:44:50.283575 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.283529    2554 factory.go:223] Registration of the systemd container factory successfully
Sep 16 18:44:50.283788 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.283774    2554 factory.go:153] Registering CRI-O factory
Sep 16 18:44:50.283788 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.283788    2554 factory.go:223] Registration of the crio container factory successfully
Sep 16 18:44:50.283930 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.283840    2554 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 16 18:44:50.283930 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.283859    2554 factory.go:103] Registering Raw factory
Sep 16 18:44:50.283930 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.283873    2554 manager.go:1196] Started watching for new ooms in manager
Sep 16 18:44:50.284211 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.284200    2554 manager.go:349] Starting recovery of all containers
Sep 16 18:44:50.285900 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.285870    2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-srr5x"
Sep 16 18:44:50.294467 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.294439    2554 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:50.296751 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.296731    2554 manager.go:354] Recovery completed
Sep 16 18:44:50.298635 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:50.298609    2554 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-241.ec2.internal\" not found" node="ip-10-0-137-241.ec2.internal"
Sep 16 18:44:50.302846 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.302786    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:50.304755 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.304740    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:50.304824 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.304771    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:50.304824 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.304781    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:50.305370 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.305346    2554 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 16 18:44:50.305370 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.305359    2554 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 16 18:44:50.305488 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.305380    2554 state_mem.go:36] "Initialized new in-memory state store"
Sep 16 18:44:50.305488 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.305361    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:50.306091 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.306075    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:50.306185 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.306111    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:50.306185 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.306126    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:50.307567 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.307553    2554 policy_none.go:49] "None policy: Start"
Sep 16 18:44:50.307630 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.307571    2554 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 16 18:44:50.307630 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.307583    2554 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 16 18:44:50.309734 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.309710    2554 policy_none.go:47] "Start"
Sep 16 18:44:50.340321 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.340295    2554 manager.go:337] "Starting Device Plugin manager"
Sep 16 18:44:50.357843 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:50.340354    2554 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 16 18:44:50.357843 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.340375    2554 server.go:87] "Starting device plugin registration server"
Sep 16 18:44:50.357843 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.340639    2554 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 16 18:44:50.357843 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.340652    2554 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 16 18:44:50.357843 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.340743    2554 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 16 18:44:50.357843 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.340816    2554 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 16 18:44:50.357843 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.340826    2554 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 16 18:44:50.357843 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:50.341291    2554 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 16 18:44:50.357843 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:50.341353    2554 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-241.ec2.internal\" not found"
Sep 16 18:44:50.436323 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.436293    2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 16 18:44:50.437375 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.437356    2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 16 18:44:50.437446 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.437417    2554 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 16 18:44:50.437496 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.437450    2554 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 16 18:44:50.437540 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:50.437493    2554 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 16 18:44:50.440256 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.440241    2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:50.440967 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.440954    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:50.441676 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.441661    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:50.441789 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.441692    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:50.441789 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.441705    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:50.441789 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.441740    2554 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-241.ec2.internal"
Sep 16 18:44:50.450212 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.450160    2554 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-241.ec2.internal"
Sep 16 18:44:50.450212 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:50.450179    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-241.ec2.internal\": node \"ip-10-0-137-241.ec2.internal\" not found"
Sep 16 18:44:50.468361 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:50.468342    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-241.ec2.internal\" not found"
Sep 16 18:44:50.538100 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.538070    2554 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-241.ec2.internal"]
Sep 16 18:44:50.538176 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.538132    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:50.538974 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.538962    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:50.539038 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.538987    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:50.539038 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.539001    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:50.541235 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.541221    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:50.541383 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.541371    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal"
Sep 16 18:44:50.541417 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.541397    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:50.541952 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.541936    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:50.541952 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.541946    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:50.542059 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.541961    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:50.542059 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.541965    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:50.542059 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.541973    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:50.542059 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.541974    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:50.544607 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.544593    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-241.ec2.internal"
Sep 16 18:44:50.544676 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.544620    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 16 18:44:50.545234 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.545220    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 16 18:44:50.545297 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.545243    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 16 18:44:50.545297 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.545253    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeHasSufficientPID"
Sep 16 18:44:50.569161 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:50.569145    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-241.ec2.internal\" not found"
Sep 16 18:44:50.569288 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:50.569272    2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-241.ec2.internal\" not found" node="ip-10-0-137-241.ec2.internal"
Sep 16 18:44:50.573429 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:50.573413    2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-241.ec2.internal\" not found" node="ip-10-0-137-241.ec2.internal"
Sep 16 18:44:50.669436 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:50.669416    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-241.ec2.internal\" not found"
Sep 16 18:44:50.684592 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.684572    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/54df781194ca8ea1f66ff331bd3b1eb5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal\" (UID: \"54df781194ca8ea1f66ff331bd3b1eb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal"
Sep 16 18:44:50.684681 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.684602    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54df781194ca8ea1f66ff331bd3b1eb5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal\" (UID: \"54df781194ca8ea1f66ff331bd3b1eb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal"
Sep 16 18:44:50.684681 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.684628    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/515eac2af796c2741f582b95e3ba4c7a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-241.ec2.internal\" (UID: \"515eac2af796c2741f582b95e3ba4c7a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-241.ec2.internal"
Sep 16 18:44:50.770096 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:50.770046    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-241.ec2.internal\" not found"
Sep 16 18:44:50.785378 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.785359    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/54df781194ca8ea1f66ff331bd3b1eb5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal\" (UID: \"54df781194ca8ea1f66ff331bd3b1eb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal"
Sep 16 18:44:50.785439 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.785411    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54df781194ca8ea1f66ff331bd3b1eb5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal\" (UID: \"54df781194ca8ea1f66ff331bd3b1eb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal"
Sep 16 18:44:50.785524 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.785382    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54df781194ca8ea1f66ff331bd3b1eb5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal\" (UID: \"54df781194ca8ea1f66ff331bd3b1eb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal"
Sep 16 18:44:50.785524 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.785455    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/54df781194ca8ea1f66ff331bd3b1eb5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal\" (UID: \"54df781194ca8ea1f66ff331bd3b1eb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal"
Sep 16 18:44:50.785524 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.785496    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/515eac2af796c2741f582b95e3ba4c7a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-241.ec2.internal\" (UID: \"515eac2af796c2741f582b95e3ba4c7a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-241.ec2.internal"
Sep 16 18:44:50.785524 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.785521    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/515eac2af796c2741f582b95e3ba4c7a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-241.ec2.internal\" (UID: \"515eac2af796c2741f582b95e3ba4c7a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-241.ec2.internal"
Sep 16 18:44:50.870517 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:50.870500    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-241.ec2.internal\" not found"
Sep 16 18:44:50.871605 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.871588    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal"
Sep 16 18:44:50.876394 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:50.876286    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-241.ec2.internal"
Sep 16 18:44:50.971403 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:50.971379    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-241.ec2.internal\" not found"
Sep 16 18:44:51.071452 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:51.071426    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-241.ec2.internal\" not found"
Sep 16 18:44:51.171653 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:51.171612    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-241.ec2.internal\" not found"
Sep 16 18:44:51.185034 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:51.185010    2554 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:51.225333 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:51.225309    2554 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 16 18:44:51.225723 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:51.225459    2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:44:51.225723 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:51.225476    2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:44:51.225723 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:51.225493    2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 16 18:44:51.272160 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:51.272132    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-241.ec2.internal\" not found"
Sep 16 18:44:51.291108 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:51.291075    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-15 18:39:51 +0000 UTC" deadline="2028-06-12 00:54:21.875762218 +0000 UTC"
Sep 16 18:44:51.291108 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:51.291103    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15222h9m30.584662779s"
Sep 16 18:44:51.373823 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:51.373766    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-241.ec2.internal\" not found"
Sep 16 18:44:51.476495 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:51.476471    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-241.ec2.internal\" not found"
Sep 16 18:44:51.497940 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:51.497919    2554 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:51.583604 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:51.583452    2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal"
Sep 16 18:44:51.595826 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:51.595799    2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:44:51.596859 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:51.596839    2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-241.ec2.internal"
Sep 16 18:44:51.606259 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:51.606237    2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 16 18:44:51.756200 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:51.756167    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod515eac2af796c2741f582b95e3ba4c7a.slice/crio-b0baaa2ce058e7f253452c5b235e177b07e6c54794cacf92a6347d74e7d7f821 WatchSource:0}: Error finding container b0baaa2ce058e7f253452c5b235e177b07e6c54794cacf92a6347d74e7d7f821: Status 404 returned error can't find the container with id b0baaa2ce058e7f253452c5b235e177b07e6c54794cacf92a6347d74e7d7f821
Sep 16 18:44:51.757887 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:51.757405    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54df781194ca8ea1f66ff331bd3b1eb5.slice/crio-6c5c5d428d5ff20a765a65fa842f3584fbe8fdbcf56088c840cc8db8397000e5 WatchSource:0}: Error finding container 6c5c5d428d5ff20a765a65fa842f3584fbe8fdbcf56088c840cc8db8397000e5: Status 404 returned error can't find the container with id 6c5c5d428d5ff20a765a65fa842f3584fbe8fdbcf56088c840cc8db8397000e5
Sep 16 18:44:51.762530 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:51.762509    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:44:52.265980 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.265952    2554 apiserver.go:52] "Watching apiserver"
Sep 16 18:44:52.272550 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.272524    2554 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 16 18:44:52.275632 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.275056    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-241.ec2.internal","openshift-cluster-node-tuning-operator/tuned-6cmnf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal","openshift-multus/multus-additional-cni-plugins-frrd7","openshift-network-operator/iptables-alerter-xr8t8","kube-system/global-pull-secret-syncer-4pmm6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjknf","openshift-dns/node-resolver-tdn77","openshift-image-registry/node-ca-grtxh","openshift-multus/multus-w7rmx","openshift-multus/network-metrics-daemon-zp95h","openshift-network-diagnostics/network-check-target-dvtnm","openshift-ovn-kubernetes/ovnkube-node-kdln4","kube-system/konnectivity-agent-k8kfm"]
Sep 16 18:44:52.278346 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.278117    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjknf"
Sep 16 18:44:52.280985 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.280371    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 16 18:44:52.280985 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.280532    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pjxsf\""
Sep 16 18:44:52.280985 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.280674    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 16 18:44:52.280985 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.280816    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 16 18:44:52.281273 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.280992    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.284316 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.284294    2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 16 18:44:52.285335 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.284300    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k8kfm"
Sep 16 18:44:52.285335 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.285131    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 16 18:44:52.285335 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.285183    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 16 18:44:52.285335 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.285230    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 16 18:44:52.285335 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.285230    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 16 18:44:52.285335 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.285288    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 16 18:44:52.285679 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.285496    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 16 18:44:52.285679 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.285616    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lk9mx\""
Sep 16 18:44:52.286187 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.286164    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-frrd7"
Sep 16 18:44:52.287230 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.287204    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9qnb6\""
Sep 16 18:44:52.287399 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.287384    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 16 18:44:52.287551 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.287531    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 16 18:44:52.287953 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.287938    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 16 18:44:52.288133 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.288115    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-96s9g\""
Sep 16 18:44:52.288462 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.288444    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 16 18:44:52.288646 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.288630    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 16 18:44:52.289558 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.289139    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 16 18:44:52.289911 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.289284    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 16 18:44:52.290413 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.289374    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xr8t8"
Sep 16 18:44:52.292064 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.292042    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:44:52.292160 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:52.292137    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pmm6" podUID="9f2230c8-6009-4227-8dbc-12e10ebe55f2"
Sep 16 18:44:52.292262 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.292237    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 16 18:44:52.293458 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.292512    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nxv64\""
Sep 16 18:44:52.293458 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.292512    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:44:52.293458 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.292626    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8f7856f8-39ee-46ec-8eb5-48dd0503e543-ovnkube-config\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.293458 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.292657    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8f7856f8-39ee-46ec-8eb5-48dd0503e543-env-overrides\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.293458 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.292684    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3b686f7c-1dab-475a-97fa-d9094746316c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gjknf\" (UID: \"3b686f7c-1dab-475a-97fa-d9094746316c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjknf"
Sep 16 18:44:52.293458 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.292724    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-run-systemd\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.293458 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.292770    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-var-lib-openvswitch\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.293458 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.292796    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-host-cni-netd\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.293458 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.292820    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-host-kubelet\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.293458 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.292855    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-etc-openvswitch\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.293458 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.292900    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-run-openvswitch\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.293458 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.292929    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-host-slash\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.293458 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.292973    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3b686f7c-1dab-475a-97fa-d9094746316c-registration-dir\") pod \"aws-ebs-csi-driver-node-gjknf\" (UID: \"3b686f7c-1dab-475a-97fa-d9094746316c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjknf"
Sep 16 18:44:52.293458 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.292982    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 16 18:44:52.293458 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.293000    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3b686f7c-1dab-475a-97fa-d9094746316c-etc-selinux\") pod \"aws-ebs-csi-driver-node-gjknf\" (UID: \"3b686f7c-1dab-475a-97fa-d9094746316c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjknf"
Sep 16 18:44:52.293458 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.293043    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-host-run-netns\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.293458 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.293068    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8f7856f8-39ee-46ec-8eb5-48dd0503e543-ovnkube-script-lib\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.293458 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.293092    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/698bb50e-290e-4842-af73-e8e8f7925f3a-system-cni-dir\") pod \"multus-additional-cni-plugins-frrd7\" (UID: \"698bb50e-290e-4842-af73-e8e8f7925f3a\") " pod="openshift-multus/multus-additional-cni-plugins-frrd7"
Sep 16 18:44:52.294465 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.293119    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/698bb50e-290e-4842-af73-e8e8f7925f3a-cnibin\") pod \"multus-additional-cni-plugins-frrd7\" (UID: \"698bb50e-290e-4842-af73-e8e8f7925f3a\") " pod="openshift-multus/multus-additional-cni-plugins-frrd7"
Sep 16 18:44:52.294465 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.293143    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/698bb50e-290e-4842-af73-e8e8f7925f3a-cni-binary-copy\") pod \"multus-additional-cni-plugins-frrd7\" (UID: \"698bb50e-290e-4842-af73-e8e8f7925f3a\") " pod="openshift-multus/multus-additional-cni-plugins-frrd7"
Sep 16 18:44:52.294465 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.293170    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/698bb50e-290e-4842-af73-e8e8f7925f3a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-frrd7\" (UID: \"698bb50e-290e-4842-af73-e8e8f7925f3a\") " pod="openshift-multus/multus-additional-cni-plugins-frrd7"
Sep 16 18:44:52.294465 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.293201    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62h9k\" (UniqueName: \"kubernetes.io/projected/698bb50e-290e-4842-af73-e8e8f7925f3a-kube-api-access-62h9k\") pod \"multus-additional-cni-plugins-frrd7\" (UID: \"698bb50e-290e-4842-af73-e8e8f7925f3a\") " pod="openshift-multus/multus-additional-cni-plugins-frrd7"
Sep 16 18:44:52.294465 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.293236    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-run-ovn\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.294465 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.293284    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-log-socket\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.294465 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.293319    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f7856f8-39ee-46ec-8eb5-48dd0503e543-ovn-node-metrics-cert\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.294465 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.293342    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/972f8c3b-0fc7-4073-be0e-d98f28705085-konnectivity-ca\") pod \"konnectivity-agent-k8kfm\" (UID: \"972f8c3b-0fc7-4073-be0e-d98f28705085\") " pod="kube-system/konnectivity-agent-k8kfm"
Sep 16 18:44:52.294465 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.293382    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/698bb50e-290e-4842-af73-e8e8f7925f3a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-frrd7\" (UID: \"698bb50e-290e-4842-af73-e8e8f7925f3a\") " pod="openshift-multus/multus-additional-cni-plugins-frrd7"
Sep 16 18:44:52.294465 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.293423    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvnmv\" (UniqueName: \"kubernetes.io/projected/3b686f7c-1dab-475a-97fa-d9094746316c-kube-api-access-rvnmv\") pod \"aws-ebs-csi-driver-node-gjknf\" (UID: \"3b686f7c-1dab-475a-97fa-d9094746316c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjknf"
Sep 16 18:44:52.294465 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.293444    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-host-run-ovn-kubernetes\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.294465 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.293475    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.294465 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.293502    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-host-cni-bin\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.294465 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.293526    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3b686f7c-1dab-475a-97fa-d9094746316c-socket-dir\") pod \"aws-ebs-csi-driver-node-gjknf\" (UID: \"3b686f7c-1dab-475a-97fa-d9094746316c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjknf"
Sep 16 18:44:52.294465 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.293550    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3b686f7c-1dab-475a-97fa-d9094746316c-device-dir\") pod \"aws-ebs-csi-driver-node-gjknf\" (UID: \"3b686f7c-1dab-475a-97fa-d9094746316c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjknf"
Sep 16 18:44:52.294465 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.293577    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3b686f7c-1dab-475a-97fa-d9094746316c-sys-fs\") pod \"aws-ebs-csi-driver-node-gjknf\" (UID: \"3b686f7c-1dab-475a-97fa-d9094746316c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjknf"
Sep 16 18:44:52.295315 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.293601    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttbwz\" (UniqueName: \"kubernetes.io/projected/8f7856f8-39ee-46ec-8eb5-48dd0503e543-kube-api-access-ttbwz\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.295315 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.293631    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/972f8c3b-0fc7-4073-be0e-d98f28705085-agent-certs\") pod \"konnectivity-agent-k8kfm\" (UID: \"972f8c3b-0fc7-4073-be0e-d98f28705085\") " pod="kube-system/konnectivity-agent-k8kfm"
Sep 16 18:44:52.295315 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.293656    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/698bb50e-290e-4842-af73-e8e8f7925f3a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-frrd7\" (UID: \"698bb50e-290e-4842-af73-e8e8f7925f3a\") " pod="openshift-multus/multus-additional-cni-plugins-frrd7"
Sep 16 18:44:52.295315 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.293693    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-systemd-units\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.295315 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.293729    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-node-log\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.295931 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.295893    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-grtxh"
Sep 16 18:44:52.297903 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.297881    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rqqgb\""
Sep 16 18:44:52.298000 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.297941    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 16 18:44:52.298000 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.297939    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 16 18:44:52.298110 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.297942    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 16 18:44:52.298839 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.298816    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tdn77"
Sep 16 18:44:52.300941 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.300907    2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 16 18:44:52.301550 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.301167    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4bfkc\""
Sep 16 18:44:52.301550 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.301273    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 16 18:44:52.301550 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.301396    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 16 18:44:52.301804 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.301662    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.303741 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.303721    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tqwrj\""
Sep 16 18:44:52.303859 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.303795    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:44:52.304021 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.304003    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 16 18:44:52.304420 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.304399    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.306173 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.306154    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-p8nvl\""
Sep 16 18:44:52.306272 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.306203    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 16 18:44:52.307250 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.307229    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:44:52.307331 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:52.307302    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zp95h" podUID="d59dba60-7c0d-4e12-b53a-3462d4b9791a"
Sep 16 18:44:52.309554 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.309532    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:44:52.309680 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:52.309597    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvtnm" podUID="b17c052d-1029-47d7-960f-af161a00f71c"
Sep 16 18:44:52.324128 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.324104    2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-b5258"
Sep 16 18:44:52.333950 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.333591    2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-b5258"
Sep 16 18:44:52.356874 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.356858    2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:52.384724 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.384660    2554 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 16 18:44:52.394881 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.394851    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-host-var-lib-cni-multus\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.394980 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.394886    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-tmp\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.394980 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.394916    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8f7856f8-39ee-46ec-8eb5-48dd0503e543-ovnkube-config\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.394980 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.394943    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3b686f7c-1dab-475a-97fa-d9094746316c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gjknf\" (UID: \"3b686f7c-1dab-475a-97fa-d9094746316c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjknf"
Sep 16 18:44:52.394980 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.394965    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-run-systemd\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.395200 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.394991    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-var-lib-openvswitch\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.395200 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395048    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-var-lib-openvswitch\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.395200 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395059    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3b686f7c-1dab-475a-97fa-d9094746316c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-gjknf\" (UID: \"3b686f7c-1dab-475a-97fa-d9094746316c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjknf"
Sep 16 18:44:52.395200 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395072    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75c1c001-b653-405b-9227-89d064e216bd-cni-binary-copy\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.395200 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395048    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-run-systemd\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.395200 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395113    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e82386e7-000e-4693-aa95-d54f59be5f8b-host\") pod \"node-ca-grtxh\" (UID: \"e82386e7-000e-4693-aa95-d54f59be5f8b\") " pod="openshift-image-registry/node-ca-grtxh"
Sep 16 18:44:52.395200 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395149    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8f7856f8-39ee-46ec-8eb5-48dd0503e543-ovnkube-script-lib\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.395200 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395190    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-etc-openvswitch\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.395650 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395220    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-host-slash\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.395650 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395250    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3b686f7c-1dab-475a-97fa-d9094746316c-registration-dir\") pod \"aws-ebs-csi-driver-node-gjknf\" (UID: \"3b686f7c-1dab-475a-97fa-d9094746316c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjknf"
Sep 16 18:44:52.395650 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395256    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-etc-openvswitch\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.395650 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395276    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmc4j\" (UniqueName: \"kubernetes.io/projected/75c1c001-b653-405b-9227-89d064e216bd-kube-api-access-gmc4j\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.395650 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395304    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/698bb50e-290e-4842-af73-e8e8f7925f3a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-frrd7\" (UID: \"698bb50e-290e-4842-af73-e8e8f7925f3a\") " pod="openshift-multus/multus-additional-cni-plugins-frrd7"
Sep 16 18:44:52.395650 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395334    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62h9k\" (UniqueName: \"kubernetes.io/projected/698bb50e-290e-4842-af73-e8e8f7925f3a-kube-api-access-62h9k\") pod \"multus-additional-cni-plugins-frrd7\" (UID: \"698bb50e-290e-4842-af73-e8e8f7925f3a\") " pod="openshift-multus/multus-additional-cni-plugins-frrd7"
Sep 16 18:44:52.395650 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395361    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-etc-kubernetes\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.395650 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395354    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3b686f7c-1dab-475a-97fa-d9094746316c-registration-dir\") pod \"aws-ebs-csi-driver-node-gjknf\" (UID: \"3b686f7c-1dab-475a-97fa-d9094746316c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjknf"
Sep 16 18:44:52.395650 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395307    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-host-slash\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.395650 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395391    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-system-cni-dir\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.395650 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395421    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-multus-socket-dir-parent\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.395650 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395446    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/18c10e7d-2ca0-4852-a696-f3ddebc29f69-iptables-alerter-script\") pod \"iptables-alerter-xr8t8\" (UID: \"18c10e7d-2ca0-4852-a696-f3ddebc29f69\") " pod="openshift-network-operator/iptables-alerter-xr8t8"
Sep 16 18:44:52.395650 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395470    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18c10e7d-2ca0-4852-a696-f3ddebc29f69-host-slash\") pod \"iptables-alerter-xr8t8\" (UID: \"18c10e7d-2ca0-4852-a696-f3ddebc29f69\") " pod="openshift-network-operator/iptables-alerter-xr8t8"
Sep 16 18:44:52.395650 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395492    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpqld\" (UniqueName: \"kubernetes.io/projected/e82386e7-000e-4693-aa95-d54f59be5f8b-kube-api-access-bpqld\") pod \"node-ca-grtxh\" (UID: \"e82386e7-000e-4693-aa95-d54f59be5f8b\") " pod="openshift-image-registry/node-ca-grtxh"
Sep 16 18:44:52.395650 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395517    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-log-socket\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.395650 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395542    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f7856f8-39ee-46ec-8eb5-48dd0503e543-ovn-node-metrics-cert\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.395650 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395563    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8f7856f8-39ee-46ec-8eb5-48dd0503e543-ovnkube-config\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.396545 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395568    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk8v5\" (UniqueName: \"kubernetes.io/projected/8f16d955-3a40-4270-8810-9f50eecea21d-kube-api-access-lk8v5\") pod \"node-resolver-tdn77\" (UID: \"8f16d955-3a40-4270-8810-9f50eecea21d\") " pod="openshift-dns/node-resolver-tdn77"
Sep 16 18:44:52.396545 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395658    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-log-socket\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.396545 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395695    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8f7856f8-39ee-46ec-8eb5-48dd0503e543-ovnkube-script-lib\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.396545 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395700    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-sys\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.396545 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395761    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-etc-tuned\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.396545 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395829    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-host-run-ovn-kubernetes\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.396545 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395862    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-hostroot\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.396545 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395888    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-host-cni-bin\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.396545 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395899    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-host-run-ovn-kubernetes\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.396545 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395918    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3b686f7c-1dab-475a-97fa-d9094746316c-socket-dir\") pod \"aws-ebs-csi-driver-node-gjknf\" (UID: \"3b686f7c-1dab-475a-97fa-d9094746316c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjknf"
Sep 16 18:44:52.396545 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395957    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3b686f7c-1dab-475a-97fa-d9094746316c-sys-fs\") pod \"aws-ebs-csi-driver-node-gjknf\" (UID: \"3b686f7c-1dab-475a-97fa-d9094746316c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjknf"
Sep 16 18:44:52.396545 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395984    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-etc-sysconfig\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.396545 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.395984    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-host-cni-bin\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.396545 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396009    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f2230c8-6009-4227-8dbc-12e10ebe55f2-original-pull-secret\") pod \"global-pull-secret-syncer-4pmm6\" (UID: \"9f2230c8-6009-4227-8dbc-12e10ebe55f2\") " pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:44:52.396545 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396019    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3b686f7c-1dab-475a-97fa-d9094746316c-socket-dir\") pod \"aws-ebs-csi-driver-node-gjknf\" (UID: \"3b686f7c-1dab-475a-97fa-d9094746316c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjknf"
Sep 16 18:44:52.396545 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396032    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-lib-modules\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.396545 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396061    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-host-run-multus-certs\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.397401 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396086    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8f16d955-3a40-4270-8810-9f50eecea21d-hosts-file\") pod \"node-resolver-tdn77\" (UID: \"8f16d955-3a40-4270-8810-9f50eecea21d\") " pod="openshift-dns/node-resolver-tdn77"
Sep 16 18:44:52.397401 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396121    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-etc-systemd\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.397401 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396064    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3b686f7c-1dab-475a-97fa-d9094746316c-sys-fs\") pod \"aws-ebs-csi-driver-node-gjknf\" (UID: \"3b686f7c-1dab-475a-97fa-d9094746316c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjknf"
Sep 16 18:44:52.397401 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396144    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-host-run-k8s-cni-cncf-io\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.397401 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396162    2554 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 16 18:44:52.397401 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396173    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8f7856f8-39ee-46ec-8eb5-48dd0503e543-env-overrides\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.397401 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396201    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-host-cni-netd\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.397401 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396226    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-multus-conf-dir\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.397401 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396278    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-host-cni-netd\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.397401 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396315    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/698bb50e-290e-4842-af73-e8e8f7925f3a-system-cni-dir\") pod \"multus-additional-cni-plugins-frrd7\" (UID: \"698bb50e-290e-4842-af73-e8e8f7925f3a\") " pod="openshift-multus/multus-additional-cni-plugins-frrd7"
Sep 16 18:44:52.397401 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396342    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87wct\" (UniqueName: \"kubernetes.io/projected/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-kube-api-access-87wct\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.397401 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396367    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-host-kubelet\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.397401 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396392    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-run-openvswitch\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.397401 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396408    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/698bb50e-290e-4842-af73-e8e8f7925f3a-system-cni-dir\") pod \"multus-additional-cni-plugins-frrd7\" (UID: \"698bb50e-290e-4842-af73-e8e8f7925f3a\") " pod="openshift-multus/multus-additional-cni-plugins-frrd7"
Sep 16 18:44:52.397401 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396423    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/698bb50e-290e-4842-af73-e8e8f7925f3a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-frrd7\" (UID: \"698bb50e-290e-4842-af73-e8e8f7925f3a\") " pod="openshift-multus/multus-additional-cni-plugins-frrd7"
Sep 16 18:44:52.397401 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396420    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3b686f7c-1dab-475a-97fa-d9094746316c-etc-selinux\") pod \"aws-ebs-csi-driver-node-gjknf\" (UID: \"3b686f7c-1dab-475a-97fa-d9094746316c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjknf"
Sep 16 18:44:52.397401 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396443    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-host-kubelet\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.398289 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396460    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-run-openvswitch\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.398289 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396470    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9f2230c8-6009-4227-8dbc-12e10ebe55f2-dbus\") pod \"global-pull-secret-syncer-4pmm6\" (UID: \"9f2230c8-6009-4227-8dbc-12e10ebe55f2\") " pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:44:52.398289 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396505    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-host-run-netns\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.398289 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396513    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3b686f7c-1dab-475a-97fa-d9094746316c-etc-selinux\") pod \"aws-ebs-csi-driver-node-gjknf\" (UID: \"3b686f7c-1dab-475a-97fa-d9094746316c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjknf"
Sep 16 18:44:52.398289 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396533    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/698bb50e-290e-4842-af73-e8e8f7925f3a-cnibin\") pod \"multus-additional-cni-plugins-frrd7\" (UID: \"698bb50e-290e-4842-af73-e8e8f7925f3a\") " pod="openshift-multus/multus-additional-cni-plugins-frrd7"
Sep 16 18:44:52.398289 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396558    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/698bb50e-290e-4842-af73-e8e8f7925f3a-cni-binary-copy\") pod \"multus-additional-cni-plugins-frrd7\" (UID: \"698bb50e-290e-4842-af73-e8e8f7925f3a\") " pod="openshift-multus/multus-additional-cni-plugins-frrd7"
Sep 16 18:44:52.398289 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396570    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-host-run-netns\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.398289 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396582    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-etc-sysctl-d\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.398289 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396584    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8f7856f8-39ee-46ec-8eb5-48dd0503e543-env-overrides\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.398289 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396588    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/698bb50e-290e-4842-af73-e8e8f7925f3a-cnibin\") pod \"multus-additional-cni-plugins-frrd7\" (UID: \"698bb50e-290e-4842-af73-e8e8f7925f3a\") " pod="openshift-multus/multus-additional-cni-plugins-frrd7"
Sep 16 18:44:52.398289 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396606    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-host\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.398289 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396630    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-var-lib-kubelet\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.398289 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396655    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-multus-cni-dir\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.398289 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396677    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-cnibin\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.398289 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396703    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-run-ovn\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.398289 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396728    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/972f8c3b-0fc7-4073-be0e-d98f28705085-konnectivity-ca\") pod \"konnectivity-agent-k8kfm\" (UID: \"972f8c3b-0fc7-4073-be0e-d98f28705085\") " pod="kube-system/konnectivity-agent-k8kfm"
Sep 16 18:44:52.398289 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396785    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-run-ovn\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.399068 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396789    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/698bb50e-290e-4842-af73-e8e8f7925f3a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-frrd7\" (UID: \"698bb50e-290e-4842-af73-e8e8f7925f3a\") " pod="openshift-multus/multus-additional-cni-plugins-frrd7"
Sep 16 18:44:52.399068 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396822    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-run\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.399068 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396848    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-host-run-netns\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.399068 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396873    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-host-var-lib-kubelet\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.399068 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396898    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/75c1c001-b653-405b-9227-89d064e216bd-multus-daemon-config\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.399068 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396924    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq4hn\" (UniqueName: \"kubernetes.io/projected/d59dba60-7c0d-4e12-b53a-3462d4b9791a-kube-api-access-zq4hn\") pod \"network-metrics-daemon-zp95h\" (UID: \"d59dba60-7c0d-4e12-b53a-3462d4b9791a\") " pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:44:52.399068 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396932    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/698bb50e-290e-4842-af73-e8e8f7925f3a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-frrd7\" (UID: \"698bb50e-290e-4842-af73-e8e8f7925f3a\") " pod="openshift-multus/multus-additional-cni-plugins-frrd7"
Sep 16 18:44:52.399068 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396953    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvnmv\" (UniqueName: \"kubernetes.io/projected/3b686f7c-1dab-475a-97fa-d9094746316c-kube-api-access-rvnmv\") pod \"aws-ebs-csi-driver-node-gjknf\" (UID: \"3b686f7c-1dab-475a-97fa-d9094746316c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjknf"
Sep 16 18:44:52.399068 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.396995    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.399068 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.397037    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-etc-modprobe-d\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.399068 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.397063    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3b686f7c-1dab-475a-97fa-d9094746316c-device-dir\") pod \"aws-ebs-csi-driver-node-gjknf\" (UID: \"3b686f7c-1dab-475a-97fa-d9094746316c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjknf"
Sep 16 18:44:52.399068 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.397073    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.399068 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.397088    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttbwz\" (UniqueName: \"kubernetes.io/projected/8f7856f8-39ee-46ec-8eb5-48dd0503e543-kube-api-access-ttbwz\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.399068 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.397036    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/698bb50e-290e-4842-af73-e8e8f7925f3a-cni-binary-copy\") pod \"multus-additional-cni-plugins-frrd7\" (UID: \"698bb50e-290e-4842-af73-e8e8f7925f3a\") " pod="openshift-multus/multus-additional-cni-plugins-frrd7"
Sep 16 18:44:52.399068 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.397114    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/972f8c3b-0fc7-4073-be0e-d98f28705085-agent-certs\") pod \"konnectivity-agent-k8kfm\" (UID: \"972f8c3b-0fc7-4073-be0e-d98f28705085\") " pod="kube-system/konnectivity-agent-k8kfm"
Sep 16 18:44:52.399068 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.397141    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/698bb50e-290e-4842-af73-e8e8f7925f3a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-frrd7\" (UID: \"698bb50e-290e-4842-af73-e8e8f7925f3a\") " pod="openshift-multus/multus-additional-cni-plugins-frrd7"
Sep 16 18:44:52.399839 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.397156    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3b686f7c-1dab-475a-97fa-d9094746316c-device-dir\") pod \"aws-ebs-csi-driver-node-gjknf\" (UID: \"3b686f7c-1dab-475a-97fa-d9094746316c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjknf"
Sep 16 18:44:52.399839 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.397166    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f16d955-3a40-4270-8810-9f50eecea21d-tmp-dir\") pod \"node-resolver-tdn77\" (UID: \"8f16d955-3a40-4270-8810-9f50eecea21d\") " pod="openshift-dns/node-resolver-tdn77"
Sep 16 18:44:52.399839 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.397192    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jn69\" (UniqueName: \"kubernetes.io/projected/b17c052d-1029-47d7-960f-af161a00f71c-kube-api-access-2jn69\") pod \"network-check-target-dvtnm\" (UID: \"b17c052d-1029-47d7-960f-af161a00f71c\") " pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:44:52.399839 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.397219    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhqds\" (UniqueName: \"kubernetes.io/projected/18c10e7d-2ca0-4852-a696-f3ddebc29f69-kube-api-access-hhqds\") pod \"iptables-alerter-xr8t8\" (UID: \"18c10e7d-2ca0-4852-a696-f3ddebc29f69\") " pod="openshift-network-operator/iptables-alerter-xr8t8"
Sep 16 18:44:52.399839 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.397247    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e82386e7-000e-4693-aa95-d54f59be5f8b-serviceca\") pod \"node-ca-grtxh\" (UID: \"e82386e7-000e-4693-aa95-d54f59be5f8b\") " pod="openshift-image-registry/node-ca-grtxh"
Sep 16 18:44:52.399839 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.397273    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-systemd-units\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.399839 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.397297    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-node-log\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.399839 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.397343    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-etc-sysctl-conf\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.399839 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.397367    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/972f8c3b-0fc7-4073-be0e-d98f28705085-konnectivity-ca\") pod \"konnectivity-agent-k8kfm\" (UID: \"972f8c3b-0fc7-4073-be0e-d98f28705085\") " pod="kube-system/konnectivity-agent-k8kfm"
Sep 16 18:44:52.399839 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.397370    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-host-var-lib-cni-bin\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.399839 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.397431    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-etc-kubernetes\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.399839 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.397459    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59dba60-7c0d-4e12-b53a-3462d4b9791a-metrics-certs\") pod \"network-metrics-daemon-zp95h\" (UID: \"d59dba60-7c0d-4e12-b53a-3462d4b9791a\") " pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:44:52.399839 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.397467    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-node-log\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.399839 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.397460    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8f7856f8-39ee-46ec-8eb5-48dd0503e543-systemd-units\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.399839 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.397484    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9f2230c8-6009-4227-8dbc-12e10ebe55f2-kubelet-config\") pod \"global-pull-secret-syncer-4pmm6\" (UID: \"9f2230c8-6009-4227-8dbc-12e10ebe55f2\") " pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:44:52.399839 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.397734    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/698bb50e-290e-4842-af73-e8e8f7925f3a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-frrd7\" (UID: \"698bb50e-290e-4842-af73-e8e8f7925f3a\") " pod="openshift-multus/multus-additional-cni-plugins-frrd7"
Sep 16 18:44:52.401505 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.401478    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f7856f8-39ee-46ec-8eb5-48dd0503e543-ovn-node-metrics-cert\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.401593 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.401584    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/972f8c3b-0fc7-4073-be0e-d98f28705085-agent-certs\") pod \"konnectivity-agent-k8kfm\" (UID: \"972f8c3b-0fc7-4073-be0e-d98f28705085\") " pod="kube-system/konnectivity-agent-k8kfm"
Sep 16 18:44:52.405575 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.405556    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62h9k\" (UniqueName: \"kubernetes.io/projected/698bb50e-290e-4842-af73-e8e8f7925f3a-kube-api-access-62h9k\") pod \"multus-additional-cni-plugins-frrd7\" (UID: \"698bb50e-290e-4842-af73-e8e8f7925f3a\") " pod="openshift-multus/multus-additional-cni-plugins-frrd7"
Sep 16 18:44:52.406081 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.406058    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvnmv\" (UniqueName: \"kubernetes.io/projected/3b686f7c-1dab-475a-97fa-d9094746316c-kube-api-access-rvnmv\") pod \"aws-ebs-csi-driver-node-gjknf\" (UID: \"3b686f7c-1dab-475a-97fa-d9094746316c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjknf"
Sep 16 18:44:52.407654 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.407631    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttbwz\" (UniqueName: \"kubernetes.io/projected/8f7856f8-39ee-46ec-8eb5-48dd0503e543-kube-api-access-ttbwz\") pod \"ovnkube-node-kdln4\" (UID: \"8f7856f8-39ee-46ec-8eb5-48dd0503e543\") " pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.446153 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.446107    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-241.ec2.internal" event={"ID":"515eac2af796c2741f582b95e3ba4c7a","Type":"ContainerStarted","Data":"b0baaa2ce058e7f253452c5b235e177b07e6c54794cacf92a6347d74e7d7f821"}
Sep 16 18:44:52.447513 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.447488    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal" event={"ID":"54df781194ca8ea1f66ff331bd3b1eb5","Type":"ContainerStarted","Data":"6c5c5d428d5ff20a765a65fa842f3584fbe8fdbcf56088c840cc8db8397000e5"}
Sep 16 18:44:52.497879 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.497852    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-lib-modules\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.497982 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.497895    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-host-run-multus-certs\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.497982 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.497922    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8f16d955-3a40-4270-8810-9f50eecea21d-hosts-file\") pod \"node-resolver-tdn77\" (UID: \"8f16d955-3a40-4270-8810-9f50eecea21d\") " pod="openshift-dns/node-resolver-tdn77"
Sep 16 18:44:52.497982 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.497945    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-etc-systemd\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.497982 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.497965    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-host-run-k8s-cni-cncf-io\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.498169 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.497988    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-host-run-multus-certs\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.498169 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498005    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-lib-modules\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.498169 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.497990    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-multus-conf-dir\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.498169 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498023    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-multus-conf-dir\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.498169 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498046    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87wct\" (UniqueName: \"kubernetes.io/projected/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-kube-api-access-87wct\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.498169 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498068    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-host-run-k8s-cni-cncf-io\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.498169 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498075    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9f2230c8-6009-4227-8dbc-12e10ebe55f2-dbus\") pod \"global-pull-secret-syncer-4pmm6\" (UID: \"9f2230c8-6009-4227-8dbc-12e10ebe55f2\") " pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:44:52.498169 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498081    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-etc-systemd\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.498169 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498105    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-etc-sysctl-d\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.498169 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498119    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8f16d955-3a40-4270-8810-9f50eecea21d-hosts-file\") pod \"node-resolver-tdn77\" (UID: \"8f16d955-3a40-4270-8810-9f50eecea21d\") " pod="openshift-dns/node-resolver-tdn77"
Sep 16 18:44:52.498169 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498129    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-host\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.498169 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498152    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-var-lib-kubelet\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.498527 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498174    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-multus-cni-dir\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.498527 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498195    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-cnibin\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.498527 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498220    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-run\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.498527 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498223    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-etc-sysctl-d\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.498527 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498243    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-host-run-netns\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.498527 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498265    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9f2230c8-6009-4227-8dbc-12e10ebe55f2-dbus\") pod \"global-pull-secret-syncer-4pmm6\" (UID: \"9f2230c8-6009-4227-8dbc-12e10ebe55f2\") " pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:44:52.498527 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498268    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-host-var-lib-kubelet\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.498527 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498285    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-multus-cni-dir\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.498527 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498315    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/75c1c001-b653-405b-9227-89d064e216bd-multus-daemon-config\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.498527 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498307    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-host-var-lib-kubelet\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.498527 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498335    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-host\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.498527 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498344    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq4hn\" (UniqueName: \"kubernetes.io/projected/d59dba60-7c0d-4e12-b53a-3462d4b9791a-kube-api-access-zq4hn\") pod \"network-metrics-daemon-zp95h\" (UID: \"d59dba60-7c0d-4e12-b53a-3462d4b9791a\") " pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:44:52.498527 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498372    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-var-lib-kubelet\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.498527 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498375    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-cnibin\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.498527 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498433    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-run\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.498527 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498433    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-host-run-netns\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.498527 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498455    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-etc-modprobe-d\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.498527 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498485    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f16d955-3a40-4270-8810-9f50eecea21d-tmp-dir\") pod \"node-resolver-tdn77\" (UID: \"8f16d955-3a40-4270-8810-9f50eecea21d\") " pod="openshift-dns/node-resolver-tdn77"
Sep 16 18:44:52.499428 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498511    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jn69\" (UniqueName: \"kubernetes.io/projected/b17c052d-1029-47d7-960f-af161a00f71c-kube-api-access-2jn69\") pod \"network-check-target-dvtnm\" (UID: \"b17c052d-1029-47d7-960f-af161a00f71c\") " pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:44:52.499428 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498537    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhqds\" (UniqueName: \"kubernetes.io/projected/18c10e7d-2ca0-4852-a696-f3ddebc29f69-kube-api-access-hhqds\") pod \"iptables-alerter-xr8t8\" (UID: \"18c10e7d-2ca0-4852-a696-f3ddebc29f69\") " pod="openshift-network-operator/iptables-alerter-xr8t8"
Sep 16 18:44:52.499428 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498560    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e82386e7-000e-4693-aa95-d54f59be5f8b-serviceca\") pod \"node-ca-grtxh\" (UID: \"e82386e7-000e-4693-aa95-d54f59be5f8b\") " pod="openshift-image-registry/node-ca-grtxh"
Sep 16 18:44:52.499428 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498630    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-etc-modprobe-d\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.499428 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498765    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-etc-sysctl-conf\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.499428 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498815    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-host-var-lib-cni-bin\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.499428 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498852    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-etc-kubernetes\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.499428 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498875    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59dba60-7c0d-4e12-b53a-3462d4b9791a-metrics-certs\") pod \"network-metrics-daemon-zp95h\" (UID: \"d59dba60-7c0d-4e12-b53a-3462d4b9791a\") " pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:44:52.499428 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498897    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9f2230c8-6009-4227-8dbc-12e10ebe55f2-kubelet-config\") pod \"global-pull-secret-syncer-4pmm6\" (UID: \"9f2230c8-6009-4227-8dbc-12e10ebe55f2\") " pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:44:52.499428 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498913    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-host-var-lib-cni-bin\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.499428 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498927    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f16d955-3a40-4270-8810-9f50eecea21d-tmp-dir\") pod \"node-resolver-tdn77\" (UID: \"8f16d955-3a40-4270-8810-9f50eecea21d\") " pod="openshift-dns/node-resolver-tdn77"
Sep 16 18:44:52.499428 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498916    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/75c1c001-b653-405b-9227-89d064e216bd-multus-daemon-config\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.499428 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498957    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-etc-sysctl-conf\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.499428 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498919    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-host-var-lib-cni-multus\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.499428 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498985    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9f2230c8-6009-4227-8dbc-12e10ebe55f2-kubelet-config\") pod \"global-pull-secret-syncer-4pmm6\" (UID: \"9f2230c8-6009-4227-8dbc-12e10ebe55f2\") " pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:44:52.499428 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.498986    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-etc-kubernetes\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.499428 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.499008    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-tmp\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.499428 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.499020    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-host-var-lib-cni-multus\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.500425 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.499041    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75c1c001-b653-405b-9227-89d064e216bd-cni-binary-copy\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.500425 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.499056    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e82386e7-000e-4693-aa95-d54f59be5f8b-serviceca\") pod \"node-ca-grtxh\" (UID: \"e82386e7-000e-4693-aa95-d54f59be5f8b\") " pod="openshift-image-registry/node-ca-grtxh"
Sep 16 18:44:52.500425 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.499069    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e82386e7-000e-4693-aa95-d54f59be5f8b-host\") pod \"node-ca-grtxh\" (UID: \"e82386e7-000e-4693-aa95-d54f59be5f8b\") " pod="openshift-image-registry/node-ca-grtxh"
Sep 16 18:44:52.500425 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:52.499101    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:52.500425 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.499110    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e82386e7-000e-4693-aa95-d54f59be5f8b-host\") pod \"node-ca-grtxh\" (UID: \"e82386e7-000e-4693-aa95-d54f59be5f8b\") " pod="openshift-image-registry/node-ca-grtxh"
Sep 16 18:44:52.500425 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.499114    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmc4j\" (UniqueName: \"kubernetes.io/projected/75c1c001-b653-405b-9227-89d064e216bd-kube-api-access-gmc4j\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.500425 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.499147    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-etc-kubernetes\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.500425 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:52.499173    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d59dba60-7c0d-4e12-b53a-3462d4b9791a-metrics-certs podName:d59dba60-7c0d-4e12-b53a-3462d4b9791a nodeName:}" failed. No retries permitted until 2026-09-16 18:44:52.999151842 +0000 UTC m=+3.074921478 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d59dba60-7c0d-4e12-b53a-3462d4b9791a-metrics-certs") pod "network-metrics-daemon-zp95h" (UID: "d59dba60-7c0d-4e12-b53a-3462d4b9791a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:52.500425 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.499218    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-system-cni-dir\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.500425 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.499231    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-etc-kubernetes\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.500425 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.499246    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-multus-socket-dir-parent\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.500425 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.499272    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/18c10e7d-2ca0-4852-a696-f3ddebc29f69-iptables-alerter-script\") pod \"iptables-alerter-xr8t8\" (UID: \"18c10e7d-2ca0-4852-a696-f3ddebc29f69\") " pod="openshift-network-operator/iptables-alerter-xr8t8"
Sep 16 18:44:52.500425 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.499288    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-system-cni-dir\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.500425 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.499312    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18c10e7d-2ca0-4852-a696-f3ddebc29f69-host-slash\") pod \"iptables-alerter-xr8t8\" (UID: \"18c10e7d-2ca0-4852-a696-f3ddebc29f69\") " pod="openshift-network-operator/iptables-alerter-xr8t8"
Sep 16 18:44:52.500425 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.499341    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpqld\" (UniqueName: \"kubernetes.io/projected/e82386e7-000e-4693-aa95-d54f59be5f8b-kube-api-access-bpqld\") pod \"node-ca-grtxh\" (UID: \"e82386e7-000e-4693-aa95-d54f59be5f8b\") " pod="openshift-image-registry/node-ca-grtxh"
Sep 16 18:44:52.500425 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.499371    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk8v5\" (UniqueName: \"kubernetes.io/projected/8f16d955-3a40-4270-8810-9f50eecea21d-kube-api-access-lk8v5\") pod \"node-resolver-tdn77\" (UID: \"8f16d955-3a40-4270-8810-9f50eecea21d\") " pod="openshift-dns/node-resolver-tdn77"
Sep 16 18:44:52.500425 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.499374    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/18c10e7d-2ca0-4852-a696-f3ddebc29f69-host-slash\") pod \"iptables-alerter-xr8t8\" (UID: \"18c10e7d-2ca0-4852-a696-f3ddebc29f69\") " pod="openshift-network-operator/iptables-alerter-xr8t8"
Sep 16 18:44:52.501322 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.499328    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-multus-socket-dir-parent\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.501322 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.499494    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-sys\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.501322 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.499526    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-etc-tuned\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.501322 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.499552    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-hostroot\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.501322 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.499549    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-sys\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.501322 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.499579    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-etc-sysconfig\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.501322 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.499601    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/75c1c001-b653-405b-9227-89d064e216bd-hostroot\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.501322 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.499604    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f2230c8-6009-4227-8dbc-12e10ebe55f2-original-pull-secret\") pod \"global-pull-secret-syncer-4pmm6\" (UID: \"9f2230c8-6009-4227-8dbc-12e10ebe55f2\") " pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:44:52.501322 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.499634    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-etc-sysconfig\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.501322 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.499654    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/75c1c001-b653-405b-9227-89d064e216bd-cni-binary-copy\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.501322 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:52.499663    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:52.501322 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:52.499727    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f2230c8-6009-4227-8dbc-12e10ebe55f2-original-pull-secret podName:9f2230c8-6009-4227-8dbc-12e10ebe55f2 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:52.99971213 +0000 UTC m=+3.075481745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9f2230c8-6009-4227-8dbc-12e10ebe55f2-original-pull-secret") pod "global-pull-secret-syncer-4pmm6" (UID: "9f2230c8-6009-4227-8dbc-12e10ebe55f2") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:52.501322 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.499824    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/18c10e7d-2ca0-4852-a696-f3ddebc29f69-iptables-alerter-script\") pod \"iptables-alerter-xr8t8\" (UID: \"18c10e7d-2ca0-4852-a696-f3ddebc29f69\") " pod="openshift-network-operator/iptables-alerter-xr8t8"
Sep 16 18:44:52.502298 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.502237    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-tmp\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.503422 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.503400    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-etc-tuned\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.510670 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:52.510650    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:52.510779 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:52.510677    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:52.510779 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:52.510691    2554 projected.go:196] Error preparing data for projected volume kube-api-access-2jn69 for pod openshift-network-diagnostics/network-check-target-dvtnm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:52.510779 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:52.510770    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b17c052d-1029-47d7-960f-af161a00f71c-kube-api-access-2jn69 podName:b17c052d-1029-47d7-960f-af161a00f71c nodeName:}" failed. No retries permitted until 2026-09-16 18:44:53.010753825 +0000 UTC m=+3.086523464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2jn69" (UniqueName: "kubernetes.io/projected/b17c052d-1029-47d7-960f-af161a00f71c-kube-api-access-2jn69") pod "network-check-target-dvtnm" (UID: "b17c052d-1029-47d7-960f-af161a00f71c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:52.513494 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.513455    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhqds\" (UniqueName: \"kubernetes.io/projected/18c10e7d-2ca0-4852-a696-f3ddebc29f69-kube-api-access-hhqds\") pod \"iptables-alerter-xr8t8\" (UID: \"18c10e7d-2ca0-4852-a696-f3ddebc29f69\") " pod="openshift-network-operator/iptables-alerter-xr8t8"
Sep 16 18:44:52.514026 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.514001    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq4hn\" (UniqueName: \"kubernetes.io/projected/d59dba60-7c0d-4e12-b53a-3462d4b9791a-kube-api-access-zq4hn\") pod \"network-metrics-daemon-zp95h\" (UID: \"d59dba60-7c0d-4e12-b53a-3462d4b9791a\") " pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:44:52.515061 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.515036    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-87wct\" (UniqueName: \"kubernetes.io/projected/c5a1899e-6f2d-4216-aaee-f8281fe6b9dd-kube-api-access-87wct\") pod \"tuned-6cmnf\" (UID: \"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd\") " pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.516119 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.515926    2554 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:52.516254 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.516236    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmc4j\" (UniqueName: \"kubernetes.io/projected/75c1c001-b653-405b-9227-89d064e216bd-kube-api-access-gmc4j\") pod \"multus-w7rmx\" (UID: \"75c1c001-b653-405b-9227-89d064e216bd\") " pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.516652 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.516632    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk8v5\" (UniqueName: \"kubernetes.io/projected/8f16d955-3a40-4270-8810-9f50eecea21d-kube-api-access-lk8v5\") pod \"node-resolver-tdn77\" (UID: \"8f16d955-3a40-4270-8810-9f50eecea21d\") " pod="openshift-dns/node-resolver-tdn77"
Sep 16 18:44:52.517067 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.517034    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpqld\" (UniqueName: \"kubernetes.io/projected/e82386e7-000e-4693-aa95-d54f59be5f8b-kube-api-access-bpqld\") pod \"node-ca-grtxh\" (UID: \"e82386e7-000e-4693-aa95-d54f59be5f8b\") " pod="openshift-image-registry/node-ca-grtxh"
Sep 16 18:44:52.596098 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.596061    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjknf"
Sep 16 18:44:52.605756 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.605711    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-k8kfm"
Sep 16 18:44:52.606574 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:52.606547    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b686f7c_1dab_475a_97fa_d9094746316c.slice/crio-74fb4a055b5d18e1655dcd926f7b8174e46d32e38129f5770f184347bc55d9a0 WatchSource:0}: Error finding container 74fb4a055b5d18e1655dcd926f7b8174e46d32e38129f5770f184347bc55d9a0: Status 404 returned error can't find the container with id 74fb4a055b5d18e1655dcd926f7b8174e46d32e38129f5770f184347bc55d9a0
Sep 16 18:44:52.616331 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.616303    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:44:52.616919 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:52.616891    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod972f8c3b_0fc7_4073_be0e_d98f28705085.slice/crio-bdfff6359a302e7ee036b7334c5554c467b3ed698708c9956423d35a1c03c523 WatchSource:0}: Error finding container bdfff6359a302e7ee036b7334c5554c467b3ed698708c9956423d35a1c03c523: Status 404 returned error can't find the container with id bdfff6359a302e7ee036b7334c5554c467b3ed698708c9956423d35a1c03c523
Sep 16 18:44:52.622109 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.622088    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-frrd7"
Sep 16 18:44:52.629013 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:52.628978    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f7856f8_39ee_46ec_8eb5_48dd0503e543.slice/crio-c5b09cec72c6fd6e63e7536a94c69ab87d7dea36c05e49e3aafe1bc6ea7c756d WatchSource:0}: Error finding container c5b09cec72c6fd6e63e7536a94c69ab87d7dea36c05e49e3aafe1bc6ea7c756d: Status 404 returned error can't find the container with id c5b09cec72c6fd6e63e7536a94c69ab87d7dea36c05e49e3aafe1bc6ea7c756d
Sep 16 18:44:52.629721 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.629702    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xr8t8"
Sep 16 18:44:52.634091 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:52.634062    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod698bb50e_290e_4842_af73_e8e8f7925f3a.slice/crio-9cf50519067ee0c34140e55b4f28d15e7ae7a67ef2c2e79b7dca5eeae2f7f9d7 WatchSource:0}: Error finding container 9cf50519067ee0c34140e55b4f28d15e7ae7a67ef2c2e79b7dca5eeae2f7f9d7: Status 404 returned error can't find the container with id 9cf50519067ee0c34140e55b4f28d15e7ae7a67ef2c2e79b7dca5eeae2f7f9d7
Sep 16 18:44:52.639352 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.639201    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-grtxh"
Sep 16 18:44:52.639466 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:52.639443    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18c10e7d_2ca0_4852_a696_f3ddebc29f69.slice/crio-6ced6ca5e4e920bf76970ecedb5296d406718a3967adf97d7c8225785b2f8322 WatchSource:0}: Error finding container 6ced6ca5e4e920bf76970ecedb5296d406718a3967adf97d7c8225785b2f8322: Status 404 returned error can't find the container with id 6ced6ca5e4e920bf76970ecedb5296d406718a3967adf97d7c8225785b2f8322
Sep 16 18:44:52.646637 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.646611    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tdn77"
Sep 16 18:44:52.650011 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:52.649979    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode82386e7_000e_4693_aa95_d54f59be5f8b.slice/crio-798b5633d108c0eea3dffe06a05d82922ffd29ee1548627357743e6cfa3f320e WatchSource:0}: Error finding container 798b5633d108c0eea3dffe06a05d82922ffd29ee1548627357743e6cfa3f320e: Status 404 returned error can't find the container with id 798b5633d108c0eea3dffe06a05d82922ffd29ee1548627357743e6cfa3f320e
Sep 16 18:44:52.653767 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.653727    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6cmnf"
Sep 16 18:44:52.656844 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:52.656788    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f16d955_3a40_4270_8810_9f50eecea21d.slice/crio-230a6ce9a6da60e597842d958a09a47195b7fe6007a857568ab68f9e39777ea2 WatchSource:0}: Error finding container 230a6ce9a6da60e597842d958a09a47195b7fe6007a857568ab68f9e39777ea2: Status 404 returned error can't find the container with id 230a6ce9a6da60e597842d958a09a47195b7fe6007a857568ab68f9e39777ea2
Sep 16 18:44:52.659823 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:52.659758    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-w7rmx"
Sep 16 18:44:52.663501 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:44:52.663474    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a1899e_6f2d_4216_aaee_f8281fe6b9dd.slice/crio-bc74ef1486c3e8911f3a75e48f33bb78b63f8250ff859df593e2420448eb5c21 WatchSource:0}: Error finding container bc74ef1486c3e8911f3a75e48f33bb78b63f8250ff859df593e2420448eb5c21: Status 404 returned error can't find the container with id bc74ef1486c3e8911f3a75e48f33bb78b63f8250ff859df593e2420448eb5c21
Sep 16 18:44:53.003413 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:53.003323    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59dba60-7c0d-4e12-b53a-3462d4b9791a-metrics-certs\") pod \"network-metrics-daemon-zp95h\" (UID: \"d59dba60-7c0d-4e12-b53a-3462d4b9791a\") " pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:44:53.003413 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:53.003391    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f2230c8-6009-4227-8dbc-12e10ebe55f2-original-pull-secret\") pod \"global-pull-secret-syncer-4pmm6\" (UID: \"9f2230c8-6009-4227-8dbc-12e10ebe55f2\") " pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:44:53.003608 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:53.003504    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:53.003608 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:53.003587    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d59dba60-7c0d-4e12-b53a-3462d4b9791a-metrics-certs podName:d59dba60-7c0d-4e12-b53a-3462d4b9791a nodeName:}" failed. No retries permitted until 2026-09-16 18:44:54.00356487 +0000 UTC m=+4.079334498 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d59dba60-7c0d-4e12-b53a-3462d4b9791a-metrics-certs") pod "network-metrics-daemon-zp95h" (UID: "d59dba60-7c0d-4e12-b53a-3462d4b9791a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:53.003707 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:53.003501    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:53.003707 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:53.003687    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f2230c8-6009-4227-8dbc-12e10ebe55f2-original-pull-secret podName:9f2230c8-6009-4227-8dbc-12e10ebe55f2 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:54.003670943 +0000 UTC m=+4.079440558 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9f2230c8-6009-4227-8dbc-12e10ebe55f2-original-pull-secret") pod "global-pull-secret-syncer-4pmm6" (UID: "9f2230c8-6009-4227-8dbc-12e10ebe55f2") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:53.104471 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:53.104433    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jn69\" (UniqueName: \"kubernetes.io/projected/b17c052d-1029-47d7-960f-af161a00f71c-kube-api-access-2jn69\") pod \"network-check-target-dvtnm\" (UID: \"b17c052d-1029-47d7-960f-af161a00f71c\") " pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:44:53.104647 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:53.104628    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:53.104710 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:53.104657    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:53.104710 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:53.104672    2554 projected.go:196] Error preparing data for projected volume kube-api-access-2jn69 for pod openshift-network-diagnostics/network-check-target-dvtnm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:53.104823 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:53.104760    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b17c052d-1029-47d7-960f-af161a00f71c-kube-api-access-2jn69 podName:b17c052d-1029-47d7-960f-af161a00f71c nodeName:}" failed. No retries permitted until 2026-09-16 18:44:54.104723476 +0000 UTC m=+4.180493114 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2jn69" (UniqueName: "kubernetes.io/projected/b17c052d-1029-47d7-960f-af161a00f71c-kube-api-access-2jn69") pod "network-check-target-dvtnm" (UID: "b17c052d-1029-47d7-960f-af161a00f71c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:53.336793 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:53.336702    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:39:53 +0000 UTC" deadline="2028-05-01 05:58:37.847252258 +0000 UTC"
Sep 16 18:44:53.336793 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:53.336755    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14219h13m44.510502603s"
Sep 16 18:44:53.438164 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:53.438130    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:44:53.438327 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:53.438233    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvtnm" podUID="b17c052d-1029-47d7-960f-af161a00f71c"
Sep 16 18:44:53.452040 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:53.452007    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tdn77" event={"ID":"8f16d955-3a40-4270-8810-9f50eecea21d","Type":"ContainerStarted","Data":"230a6ce9a6da60e597842d958a09a47195b7fe6007a857568ab68f9e39777ea2"}
Sep 16 18:44:53.453352 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:53.453310    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-grtxh" event={"ID":"e82386e7-000e-4693-aa95-d54f59be5f8b","Type":"ContainerStarted","Data":"798b5633d108c0eea3dffe06a05d82922ffd29ee1548627357743e6cfa3f320e"}
Sep 16 18:44:53.454798 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:53.454768    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xr8t8" event={"ID":"18c10e7d-2ca0-4852-a696-f3ddebc29f69","Type":"ContainerStarted","Data":"6ced6ca5e4e920bf76970ecedb5296d406718a3967adf97d7c8225785b2f8322"}
Sep 16 18:44:53.455334 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:53.455313    2554 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 16 18:44:53.456095 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:53.456070    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frrd7" event={"ID":"698bb50e-290e-4842-af73-e8e8f7925f3a","Type":"ContainerStarted","Data":"9cf50519067ee0c34140e55b4f28d15e7ae7a67ef2c2e79b7dca5eeae2f7f9d7"}
Sep 16 18:44:53.457350 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:53.457323    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdln4" event={"ID":"8f7856f8-39ee-46ec-8eb5-48dd0503e543","Type":"ContainerStarted","Data":"c5b09cec72c6fd6e63e7536a94c69ab87d7dea36c05e49e3aafe1bc6ea7c756d"}
Sep 16 18:44:53.458554 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:53.458536    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjknf" event={"ID":"3b686f7c-1dab-475a-97fa-d9094746316c","Type":"ContainerStarted","Data":"74fb4a055b5d18e1655dcd926f7b8174e46d32e38129f5770f184347bc55d9a0"}
Sep 16 18:44:53.459769 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:53.459707    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6cmnf" event={"ID":"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd","Type":"ContainerStarted","Data":"bc74ef1486c3e8911f3a75e48f33bb78b63f8250ff859df593e2420448eb5c21"}
Sep 16 18:44:53.460968 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:53.460947    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k8kfm" event={"ID":"972f8c3b-0fc7-4073-be0e-d98f28705085","Type":"ContainerStarted","Data":"bdfff6359a302e7ee036b7334c5554c467b3ed698708c9956423d35a1c03c523"}
Sep 16 18:44:54.013102 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:54.012109    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59dba60-7c0d-4e12-b53a-3462d4b9791a-metrics-certs\") pod \"network-metrics-daemon-zp95h\" (UID: \"d59dba60-7c0d-4e12-b53a-3462d4b9791a\") " pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:44:54.013102 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:54.012178    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f2230c8-6009-4227-8dbc-12e10ebe55f2-original-pull-secret\") pod \"global-pull-secret-syncer-4pmm6\" (UID: \"9f2230c8-6009-4227-8dbc-12e10ebe55f2\") " pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:44:54.013102 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:54.012300    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:54.013102 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:54.012364    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f2230c8-6009-4227-8dbc-12e10ebe55f2-original-pull-secret podName:9f2230c8-6009-4227-8dbc-12e10ebe55f2 nodeName:}" failed. No retries permitted until 2026-09-16 18:44:56.012344594 +0000 UTC m=+6.088114221 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9f2230c8-6009-4227-8dbc-12e10ebe55f2-original-pull-secret") pod "global-pull-secret-syncer-4pmm6" (UID: "9f2230c8-6009-4227-8dbc-12e10ebe55f2") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:54.013102 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:54.013019    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:54.013102 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:54.013075    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d59dba60-7c0d-4e12-b53a-3462d4b9791a-metrics-certs podName:d59dba60-7c0d-4e12-b53a-3462d4b9791a nodeName:}" failed. No retries permitted until 2026-09-16 18:44:56.013057874 +0000 UTC m=+6.088827511 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d59dba60-7c0d-4e12-b53a-3462d4b9791a-metrics-certs") pod "network-metrics-daemon-zp95h" (UID: "d59dba60-7c0d-4e12-b53a-3462d4b9791a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:54.113602 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:54.113547    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jn69\" (UniqueName: \"kubernetes.io/projected/b17c052d-1029-47d7-960f-af161a00f71c-kube-api-access-2jn69\") pod \"network-check-target-dvtnm\" (UID: \"b17c052d-1029-47d7-960f-af161a00f71c\") " pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:44:54.114214 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:54.113748    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:54.114214 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:54.113787    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:54.114214 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:54.113801    2554 projected.go:196] Error preparing data for projected volume kube-api-access-2jn69 for pod openshift-network-diagnostics/network-check-target-dvtnm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:54.114214 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:54.113862    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b17c052d-1029-47d7-960f-af161a00f71c-kube-api-access-2jn69 podName:b17c052d-1029-47d7-960f-af161a00f71c nodeName:}" failed. No retries permitted until 2026-09-16 18:44:56.113842172 +0000 UTC m=+6.189611802 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2jn69" (UniqueName: "kubernetes.io/projected/b17c052d-1029-47d7-960f-af161a00f71c-kube-api-access-2jn69") pod "network-check-target-dvtnm" (UID: "b17c052d-1029-47d7-960f-af161a00f71c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:54.339805 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:54.339683    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-15 18:39:53 +0000 UTC" deadline="2028-03-10 22:29:07.877472324 +0000 UTC"
Sep 16 18:44:54.339805 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:54.339745    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12987h44m13.537731856s"
Sep 16 18:44:54.439817 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:54.438018    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:44:54.439817 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:54.438149    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pmm6" podUID="9f2230c8-6009-4227-8dbc-12e10ebe55f2"
Sep 16 18:44:54.439817 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:54.438587    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:44:54.439817 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:54.438701    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zp95h" podUID="d59dba60-7c0d-4e12-b53a-3462d4b9791a"
Sep 16 18:44:54.474160 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:54.474125    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w7rmx" event={"ID":"75c1c001-b653-405b-9227-89d064e216bd","Type":"ContainerStarted","Data":"0adcc43aec8d347c6c9be00a2fdc1261626741dbb563e7d722ceb778a26eefab"}
Sep 16 18:44:54.486660 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:54.486630    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-241.ec2.internal" event={"ID":"515eac2af796c2741f582b95e3ba4c7a","Type":"ContainerStarted","Data":"aa2dfbcd7bfb459870044a94912f7c53b148136bc5ee965739a284072d5793ac"}
Sep 16 18:44:55.438436 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:55.437947    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:44:55.438436 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:55.438071    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvtnm" podUID="b17c052d-1029-47d7-960f-af161a00f71c"
Sep 16 18:44:55.518016 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:55.517433    2554 generic.go:356] "Generic (PLEG): container finished" podID="54df781194ca8ea1f66ff331bd3b1eb5" containerID="ac49c3ec78865e8023e6cf3a8e3a732166b387a3b2abd1011ab73023598ac1b2" exitCode=0
Sep 16 18:44:55.518016 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:55.517595    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal" event={"ID":"54df781194ca8ea1f66ff331bd3b1eb5","Type":"ContainerDied","Data":"ac49c3ec78865e8023e6cf3a8e3a732166b387a3b2abd1011ab73023598ac1b2"}
Sep 16 18:44:55.535404 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:55.534369    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-241.ec2.internal" podStartSLOduration=3.5343493329999998 podStartE2EDuration="3.534349333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:54.515609026 +0000 UTC m=+4.591378667" watchObservedRunningTime="2026-09-16 18:44:55.534349333 +0000 UTC m=+5.610118970"
Sep 16 18:44:56.031806 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:56.031702    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59dba60-7c0d-4e12-b53a-3462d4b9791a-metrics-certs\") pod \"network-metrics-daemon-zp95h\" (UID: \"d59dba60-7c0d-4e12-b53a-3462d4b9791a\") " pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:44:56.031806 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:56.031801    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f2230c8-6009-4227-8dbc-12e10ebe55f2-original-pull-secret\") pod \"global-pull-secret-syncer-4pmm6\" (UID: \"9f2230c8-6009-4227-8dbc-12e10ebe55f2\") " pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:44:56.032074 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:56.031948    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:56.032074 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:56.032034    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f2230c8-6009-4227-8dbc-12e10ebe55f2-original-pull-secret podName:9f2230c8-6009-4227-8dbc-12e10ebe55f2 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:00.031994105 +0000 UTC m=+10.107763733 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9f2230c8-6009-4227-8dbc-12e10ebe55f2-original-pull-secret") pod "global-pull-secret-syncer-4pmm6" (UID: "9f2230c8-6009-4227-8dbc-12e10ebe55f2") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:44:56.032201 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:56.032135    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:56.032201 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:56.032179    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d59dba60-7c0d-4e12-b53a-3462d4b9791a-metrics-certs podName:d59dba60-7c0d-4e12-b53a-3462d4b9791a nodeName:}" failed. No retries permitted until 2026-09-16 18:45:00.032165908 +0000 UTC m=+10.107935524 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d59dba60-7c0d-4e12-b53a-3462d4b9791a-metrics-certs") pod "network-metrics-daemon-zp95h" (UID: "d59dba60-7c0d-4e12-b53a-3462d4b9791a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:44:56.133139 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:56.133071    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jn69\" (UniqueName: \"kubernetes.io/projected/b17c052d-1029-47d7-960f-af161a00f71c-kube-api-access-2jn69\") pod \"network-check-target-dvtnm\" (UID: \"b17c052d-1029-47d7-960f-af161a00f71c\") " pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:44:56.133323 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:56.133284    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:44:56.133323 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:56.133302    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:44:56.133323 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:56.133316    2554 projected.go:196] Error preparing data for projected volume kube-api-access-2jn69 for pod openshift-network-diagnostics/network-check-target-dvtnm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:56.133516 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:56.133369    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b17c052d-1029-47d7-960f-af161a00f71c-kube-api-access-2jn69 podName:b17c052d-1029-47d7-960f-af161a00f71c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:00.133352158 +0000 UTC m=+10.209121772 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2jn69" (UniqueName: "kubernetes.io/projected/b17c052d-1029-47d7-960f-af161a00f71c-kube-api-access-2jn69") pod "network-check-target-dvtnm" (UID: "b17c052d-1029-47d7-960f-af161a00f71c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:44:56.437820 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:56.437731    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:44:56.437971 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:56.437882    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pmm6" podUID="9f2230c8-6009-4227-8dbc-12e10ebe55f2"
Sep 16 18:44:56.438274 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:56.438251    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:44:56.438429 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:56.438376    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zp95h" podUID="d59dba60-7c0d-4e12-b53a-3462d4b9791a"
Sep 16 18:44:56.523229 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:56.523192    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal" event={"ID":"54df781194ca8ea1f66ff331bd3b1eb5","Type":"ContainerStarted","Data":"8b96dc80e33ae2a042e3695fa02a5ee4d261ca03539be1b8e7a9942623f7c988"}
Sep 16 18:44:56.538974 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:56.538923    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-241.ec2.internal" podStartSLOduration=4.538903711 podStartE2EDuration="4.538903711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:44:56.53887791 +0000 UTC m=+6.614647545" watchObservedRunningTime="2026-09-16 18:44:56.538903711 +0000 UTC m=+6.614673348"
Sep 16 18:44:57.437881 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:57.437852    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:44:57.438055 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:57.437989    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvtnm" podUID="b17c052d-1029-47d7-960f-af161a00f71c"
Sep 16 18:44:58.438860 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:58.438825    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:44:58.439363 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:58.438964    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pmm6" podUID="9f2230c8-6009-4227-8dbc-12e10ebe55f2"
Sep 16 18:44:58.439499 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:58.439478    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:44:58.439629 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:58.439607    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zp95h" podUID="d59dba60-7c0d-4e12-b53a-3462d4b9791a"
Sep 16 18:44:59.438846 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:44:59.438339    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:44:59.438846 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:44:59.438470    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvtnm" podUID="b17c052d-1029-47d7-960f-af161a00f71c"
Sep 16 18:45:00.067288 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:00.067185    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59dba60-7c0d-4e12-b53a-3462d4b9791a-metrics-certs\") pod \"network-metrics-daemon-zp95h\" (UID: \"d59dba60-7c0d-4e12-b53a-3462d4b9791a\") " pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:45:00.067288 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:00.067259    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f2230c8-6009-4227-8dbc-12e10ebe55f2-original-pull-secret\") pod \"global-pull-secret-syncer-4pmm6\" (UID: \"9f2230c8-6009-4227-8dbc-12e10ebe55f2\") " pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:45:00.067531 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:00.067367    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:00.067531 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:00.067397    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:00.067531 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:00.067441    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d59dba60-7c0d-4e12-b53a-3462d4b9791a-metrics-certs podName:d59dba60-7c0d-4e12-b53a-3462d4b9791a nodeName:}" failed. No retries permitted until 2026-09-16 18:45:08.067418521 +0000 UTC m=+18.143188150 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d59dba60-7c0d-4e12-b53a-3462d4b9791a-metrics-certs") pod "network-metrics-daemon-zp95h" (UID: "d59dba60-7c0d-4e12-b53a-3462d4b9791a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:00.067531 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:00.067463    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f2230c8-6009-4227-8dbc-12e10ebe55f2-original-pull-secret podName:9f2230c8-6009-4227-8dbc-12e10ebe55f2 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:08.067451927 +0000 UTC m=+18.143221580 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9f2230c8-6009-4227-8dbc-12e10ebe55f2-original-pull-secret") pod "global-pull-secret-syncer-4pmm6" (UID: "9f2230c8-6009-4227-8dbc-12e10ebe55f2") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:00.168122 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:00.168088    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jn69\" (UniqueName: \"kubernetes.io/projected/b17c052d-1029-47d7-960f-af161a00f71c-kube-api-access-2jn69\") pod \"network-check-target-dvtnm\" (UID: \"b17c052d-1029-47d7-960f-af161a00f71c\") " pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:45:00.168293 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:00.168259    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:00.168293 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:00.168291    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:00.168416 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:00.168305    2554 projected.go:196] Error preparing data for projected volume kube-api-access-2jn69 for pod openshift-network-diagnostics/network-check-target-dvtnm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:00.168416 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:00.168371    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b17c052d-1029-47d7-960f-af161a00f71c-kube-api-access-2jn69 podName:b17c052d-1029-47d7-960f-af161a00f71c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:08.16835114 +0000 UTC m=+18.244120778 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2jn69" (UniqueName: "kubernetes.io/projected/b17c052d-1029-47d7-960f-af161a00f71c-kube-api-access-2jn69") pod "network-check-target-dvtnm" (UID: "b17c052d-1029-47d7-960f-af161a00f71c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:00.439815 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:00.439150    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:45:00.439815 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:00.439183    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:45:00.439815 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:00.439298    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zp95h" podUID="d59dba60-7c0d-4e12-b53a-3462d4b9791a"
Sep 16 18:45:00.439815 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:00.439390    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pmm6" podUID="9f2230c8-6009-4227-8dbc-12e10ebe55f2"
Sep 16 18:45:01.439007 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:01.438572    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:45:01.439007 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:01.438688    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvtnm" podUID="b17c052d-1029-47d7-960f-af161a00f71c"
Sep 16 18:45:02.438330 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:02.438296    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:45:02.438854 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:02.438485    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pmm6" podUID="9f2230c8-6009-4227-8dbc-12e10ebe55f2"
Sep 16 18:45:02.438854 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:02.438533    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:45:02.438854 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:02.438690    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zp95h" podUID="d59dba60-7c0d-4e12-b53a-3462d4b9791a"
Sep 16 18:45:03.438270 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:03.438235    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:45:03.438462 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:03.438334    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvtnm" podUID="b17c052d-1029-47d7-960f-af161a00f71c"
Sep 16 18:45:04.438294 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:04.438211    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:45:04.438437 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:04.438219    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:45:04.438437 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:04.438338    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pmm6" podUID="9f2230c8-6009-4227-8dbc-12e10ebe55f2"
Sep 16 18:45:04.438437 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:04.438406    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zp95h" podUID="d59dba60-7c0d-4e12-b53a-3462d4b9791a"
Sep 16 18:45:05.437820 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:05.437786    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:45:05.437988 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:05.437911    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvtnm" podUID="b17c052d-1029-47d7-960f-af161a00f71c"
Sep 16 18:45:06.438307 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:06.438280    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:45:06.438775 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:06.438404    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pmm6" podUID="9f2230c8-6009-4227-8dbc-12e10ebe55f2"
Sep 16 18:45:06.438775 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:06.438492    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:45:06.438775 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:06.438616    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zp95h" podUID="d59dba60-7c0d-4e12-b53a-3462d4b9791a"
Sep 16 18:45:07.438160 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:07.438135    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:45:07.438327 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:07.438251    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvtnm" podUID="b17c052d-1029-47d7-960f-af161a00f71c"
Sep 16 18:45:08.132552 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:08.132517    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f2230c8-6009-4227-8dbc-12e10ebe55f2-original-pull-secret\") pod \"global-pull-secret-syncer-4pmm6\" (UID: \"9f2230c8-6009-4227-8dbc-12e10ebe55f2\") " pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:45:08.132762 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:08.132590    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59dba60-7c0d-4e12-b53a-3462d4b9791a-metrics-certs\") pod \"network-metrics-daemon-zp95h\" (UID: \"d59dba60-7c0d-4e12-b53a-3462d4b9791a\") " pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:45:08.132762 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:08.132674    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:08.132762 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:08.132694    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:08.132762 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:08.132751    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f2230c8-6009-4227-8dbc-12e10ebe55f2-original-pull-secret podName:9f2230c8-6009-4227-8dbc-12e10ebe55f2 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:24.132733259 +0000 UTC m=+34.208502886 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9f2230c8-6009-4227-8dbc-12e10ebe55f2-original-pull-secret") pod "global-pull-secret-syncer-4pmm6" (UID: "9f2230c8-6009-4227-8dbc-12e10ebe55f2") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:08.132941 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:08.132767    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d59dba60-7c0d-4e12-b53a-3462d4b9791a-metrics-certs podName:d59dba60-7c0d-4e12-b53a-3462d4b9791a nodeName:}" failed. No retries permitted until 2026-09-16 18:45:24.132760247 +0000 UTC m=+34.208529856 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d59dba60-7c0d-4e12-b53a-3462d4b9791a-metrics-certs") pod "network-metrics-daemon-zp95h" (UID: "d59dba60-7c0d-4e12-b53a-3462d4b9791a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:08.233089 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:08.233055    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jn69\" (UniqueName: \"kubernetes.io/projected/b17c052d-1029-47d7-960f-af161a00f71c-kube-api-access-2jn69\") pod \"network-check-target-dvtnm\" (UID: \"b17c052d-1029-47d7-960f-af161a00f71c\") " pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:45:08.233240 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:08.233200    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:08.233240 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:08.233219    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:08.233240 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:08.233233    2554 projected.go:196] Error preparing data for projected volume kube-api-access-2jn69 for pod openshift-network-diagnostics/network-check-target-dvtnm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:08.233399 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:08.233281    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b17c052d-1029-47d7-960f-af161a00f71c-kube-api-access-2jn69 podName:b17c052d-1029-47d7-960f-af161a00f71c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:24.233268122 +0000 UTC m=+34.309037732 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2jn69" (UniqueName: "kubernetes.io/projected/b17c052d-1029-47d7-960f-af161a00f71c-kube-api-access-2jn69") pod "network-check-target-dvtnm" (UID: "b17c052d-1029-47d7-960f-af161a00f71c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:08.437775 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:08.437686    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:45:08.437775 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:08.437714    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:45:08.437970 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:08.437835    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zp95h" podUID="d59dba60-7c0d-4e12-b53a-3462d4b9791a"
Sep 16 18:45:08.437970 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:08.437947    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pmm6" podUID="9f2230c8-6009-4227-8dbc-12e10ebe55f2"
Sep 16 18:45:09.438652 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:09.438607    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:45:09.439122 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:09.438742    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvtnm" podUID="b17c052d-1029-47d7-960f-af161a00f71c"
Sep 16 18:45:10.439494 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:10.439457    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:45:10.439950 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:10.439575    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pmm6" podUID="9f2230c8-6009-4227-8dbc-12e10ebe55f2"
Sep 16 18:45:10.439950 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:10.439631    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:45:10.439950 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:10.439786    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zp95h" podUID="d59dba60-7c0d-4e12-b53a-3462d4b9791a"
Sep 16 18:45:11.438539 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:11.438513    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:45:11.438656 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:11.438602    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvtnm" podUID="b17c052d-1029-47d7-960f-af161a00f71c"
Sep 16 18:45:12.438333 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:12.438013    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:45:12.439161 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:12.438093    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:45:12.439161 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:12.438442    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pmm6" podUID="9f2230c8-6009-4227-8dbc-12e10ebe55f2"
Sep 16 18:45:12.439161 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:12.438489    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zp95h" podUID="d59dba60-7c0d-4e12-b53a-3462d4b9791a"
Sep 16 18:45:12.555536 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:12.555479    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovn-acl-logging/0.log"
Sep 16 18:45:12.555846 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:12.555827    2554 generic.go:356] "Generic (PLEG): container finished" podID="8f7856f8-39ee-46ec-8eb5-48dd0503e543" containerID="fb0ee08f628b300ba84655f10dc7f62a711bf7713d7ccdafb46d1dc9d7155d51" exitCode=1
Sep 16 18:45:12.555924 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:12.555893    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdln4" event={"ID":"8f7856f8-39ee-46ec-8eb5-48dd0503e543","Type":"ContainerStarted","Data":"758cbe529406df37a50b5710e84101168956b91044807a064716f76e3347e1df"}
Sep 16 18:45:12.555976 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:12.555954    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdln4" event={"ID":"8f7856f8-39ee-46ec-8eb5-48dd0503e543","Type":"ContainerStarted","Data":"d5386eb8cc2fd227a1264add9d5d93dbc1c0e5fc81328ee2f6ef158b14cb7415"}
Sep 16 18:45:12.556034 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:12.555983    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdln4" event={"ID":"8f7856f8-39ee-46ec-8eb5-48dd0503e543","Type":"ContainerStarted","Data":"04d14efa03774cb4dbba651563a878f07ec7150429d6212b67a742d67c556271"}
Sep 16 18:45:12.556034 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:12.556012    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdln4" event={"ID":"8f7856f8-39ee-46ec-8eb5-48dd0503e543","Type":"ContainerStarted","Data":"aa476bc680366371536a976709bce930156380e01176d64b8f051af9532d9657"}
Sep 16 18:45:12.556103 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:12.556044    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdln4" event={"ID":"8f7856f8-39ee-46ec-8eb5-48dd0503e543","Type":"ContainerDied","Data":"fb0ee08f628b300ba84655f10dc7f62a711bf7713d7ccdafb46d1dc9d7155d51"}
Sep 16 18:45:12.556103 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:12.556080    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdln4" event={"ID":"8f7856f8-39ee-46ec-8eb5-48dd0503e543","Type":"ContainerStarted","Data":"4c14abf5c57167e3f8bbc495db517bec3188fbf3d9cbe4f93c4058c634e360a6"}
Sep 16 18:45:12.559970 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:12.559947    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjknf" event={"ID":"3b686f7c-1dab-475a-97fa-d9094746316c","Type":"ContainerStarted","Data":"b63d2a02b7a16beae47f7d8f3a03437fdf6e8c8eba86201f28ef264a7249ec27"}
Sep 16 18:45:12.561137 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:12.561108    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6cmnf" event={"ID":"c5a1899e-6f2d-4216-aaee-f8281fe6b9dd","Type":"ContainerStarted","Data":"52bf4f3a83fe28a953621b84348a3dd85d46233e6ba902a7c141621e74e44622"}
Sep 16 18:45:12.562327 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:12.562307    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-k8kfm" event={"ID":"972f8c3b-0fc7-4073-be0e-d98f28705085","Type":"ContainerStarted","Data":"403f57b1d9f16b14c6fa31caf128836509ac4bba793eb0941649fd29ccc75b59"}
Sep 16 18:45:12.563536 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:12.563515    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-w7rmx" event={"ID":"75c1c001-b653-405b-9227-89d064e216bd","Type":"ContainerStarted","Data":"e8f8827f8b53a3937ab7d77f7b6ae4c26170ce8069e3cd5c24e29452e277b58d"}
Sep 16 18:45:12.564792 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:12.564773    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tdn77" event={"ID":"8f16d955-3a40-4270-8810-9f50eecea21d","Type":"ContainerStarted","Data":"468cf6fd997b9eaf6bc91d56cbf0649ea18ee4bd0ac6f3d737958ba2701e4f8e"}
Sep 16 18:45:12.566198 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:12.566181    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-grtxh" event={"ID":"e82386e7-000e-4693-aa95-d54f59be5f8b","Type":"ContainerStarted","Data":"3ebe81c58f879cb2869c441f23c3d063886278d10f8d7035fe76b3a216c4028b"}
Sep 16 18:45:12.567622 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:12.567604    2554 generic.go:356] "Generic (PLEG): container finished" podID="698bb50e-290e-4842-af73-e8e8f7925f3a" containerID="1c36f93df8ed3b32fb914dae5f5966c313816232b29001726d1df4920fb3e828" exitCode=0
Sep 16 18:45:12.567700 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:12.567628    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frrd7" event={"ID":"698bb50e-290e-4842-af73-e8e8f7925f3a","Type":"ContainerDied","Data":"1c36f93df8ed3b32fb914dae5f5966c313816232b29001726d1df4920fb3e828"}
Sep 16 18:45:12.638274 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:12.638230    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6cmnf" podStartSLOduration=2.849807101 podStartE2EDuration="21.638217921s" totalImagesPullingTime="18.78841082s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:12.583028577 +0000 UTC m=+22.658798208" watchObservedRunningTime="2026-09-16 18:45:12.638217921 +0000 UTC m=+22.713987599"
Sep 16 18:45:12.670758 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:12.670702    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-grtxh" podStartSLOduration=2.888190247 podStartE2EDuration="21.670693256s" totalImagesPullingTime="18.782503009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:12.670655355 +0000 UTC m=+22.746424987" watchObservedRunningTime="2026-09-16 18:45:12.670693256 +0000 UTC m=+22.746462887"
Sep 16 18:45:12.749640 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:12.749611    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-tdn77" podStartSLOduration=3.198933171 podStartE2EDuration="21.749601011s" totalImagesPullingTime="18.55066784s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:12.708134014 +0000 UTC m=+22.783903645" watchObservedRunningTime="2026-09-16 18:45:12.749601011 +0000 UTC m=+22.825370643"
Sep 16 18:45:12.769345 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:12.769313    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-w7rmx" podStartSLOduration=3.772496507 podStartE2EDuration="21.769301914s" totalImagesPullingTime="17.996805407s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:12.749942078 +0000 UTC m=+22.825711710" watchObservedRunningTime="2026-09-16 18:45:12.769301914 +0000 UTC m=+22.845071545"
Sep 16 18:45:13.032029 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:13.032009    2554 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 16 18:45:13.360819 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:13.360651    2554 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-16T18:45:13.032025833Z","UUID":"d8236b98-3685-4c52-bd3b-9b34f9bb35be","Handler":null,"Name":"","Endpoint":""}
Sep 16 18:45:13.362899 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:13.362877    2554 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 16 18:45:13.363030 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:13.362908    2554 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 16 18:45:13.438186 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:13.438161    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:45:13.438328 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:13.438271    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvtnm" podUID="b17c052d-1029-47d7-960f-af161a00f71c"
Sep 16 18:45:13.572673 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:13.572641    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xr8t8" event={"ID":"18c10e7d-2ca0-4852-a696-f3ddebc29f69","Type":"ContainerStarted","Data":"425536245fde62322055b92ca4be27b3b01d58b31a8dbf1b3c38c9583a79b9d2"}
Sep 16 18:45:13.575223 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:13.575193    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjknf" event={"ID":"3b686f7c-1dab-475a-97fa-d9094746316c","Type":"ContainerStarted","Data":"7190242a61e80d21ff276d01a00afe2eea8ec2a5588ba0213b8e5eacb069f04f"}
Sep 16 18:45:13.594852 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:13.594805    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-k8kfm" podStartSLOduration=3.7604338200000003 podStartE2EDuration="22.594790127s" totalImagesPullingTime="18.834356307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:12.769186404 +0000 UTC m=+22.844956037" watchObservedRunningTime="2026-09-16 18:45:13.594790127 +0000 UTC m=+23.670559760"
Sep 16 18:45:13.595287 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:13.595240    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xr8t8" podStartSLOduration=7.8383410510000004 podStartE2EDuration="22.595231636s" totalImagesPullingTime="14.756890585s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:13.594337131 +0000 UTC m=+23.670106764" watchObservedRunningTime="2026-09-16 18:45:13.595231636 +0000 UTC m=+23.671001261"
Sep 16 18:45:14.438630 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:14.438556    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:45:14.438832 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:14.438571    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:45:14.438832 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:14.438689    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pmm6" podUID="9f2230c8-6009-4227-8dbc-12e10ebe55f2"
Sep 16 18:45:14.438832 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:14.438780    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zp95h" podUID="d59dba60-7c0d-4e12-b53a-3462d4b9791a"
Sep 16 18:45:14.581289 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:14.581265    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovn-acl-logging/0.log"
Sep 16 18:45:14.581771 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:14.581748    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdln4" event={"ID":"8f7856f8-39ee-46ec-8eb5-48dd0503e543","Type":"ContainerStarted","Data":"7501f3fcc23b768da7dd80e319018d0a1376d5535c09f06768389ec7dd16d53d"}
Sep 16 18:45:14.584161 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:14.584128    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjknf" event={"ID":"3b686f7c-1dab-475a-97fa-d9094746316c","Type":"ContainerStarted","Data":"29002d5e881a604b913593bef17a22836f617a61b8a2da9ec24e3d03865b290b"}
Sep 16 18:45:15.438092 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:15.438066    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:45:15.438247 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:15.438156    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvtnm" podUID="b17c052d-1029-47d7-960f-af161a00f71c"
Sep 16 18:45:16.438321 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:16.438085    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:45:16.438771 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:16.438166    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:45:16.438771 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:16.438428    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pmm6" podUID="9f2230c8-6009-4227-8dbc-12e10ebe55f2"
Sep 16 18:45:16.438771 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:16.438483    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zp95h" podUID="d59dba60-7c0d-4e12-b53a-3462d4b9791a"
Sep 16 18:45:17.411616 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:17.411530    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-k8kfm"
Sep 16 18:45:17.412294 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:17.412269    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-k8kfm"
Sep 16 18:45:17.429149 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:17.429101    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-gjknf" podStartSLOduration=5.420142074 podStartE2EDuration="26.429085159s" totalImagesPullingTime="21.008943085s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:14.606243442 +0000 UTC m=+24.682013074" watchObservedRunningTime="2026-09-16 18:45:17.429085159 +0000 UTC m=+27.504854802"
Sep 16 18:45:17.438191 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:17.438167    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:45:17.438276 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:17.438256    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvtnm" podUID="b17c052d-1029-47d7-960f-af161a00f71c"
Sep 16 18:45:17.593532 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:17.593499    2554 generic.go:356] "Generic (PLEG): container finished" podID="698bb50e-290e-4842-af73-e8e8f7925f3a" containerID="36f409ea2e1fb28d1876d30a1537d2915e8fa3432e040826e2c49f61b44815ac" exitCode=0
Sep 16 18:45:17.594319 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:17.593588    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frrd7" event={"ID":"698bb50e-290e-4842-af73-e8e8f7925f3a","Type":"ContainerDied","Data":"36f409ea2e1fb28d1876d30a1537d2915e8fa3432e040826e2c49f61b44815ac"}
Sep 16 18:45:17.598309 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:17.598284    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovn-acl-logging/0.log"
Sep 16 18:45:17.598768 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:17.598743    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdln4" event={"ID":"8f7856f8-39ee-46ec-8eb5-48dd0503e543","Type":"ContainerStarted","Data":"85ad42a31582419a743e2db4b5269fb94c815ea12f4e6cef55c2592e8702357c"}
Sep 16 18:45:17.599001 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:17.598979    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-k8kfm"
Sep 16 18:45:17.599228 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:17.599206    2554 scope.go:117] "RemoveContainer" containerID="fb0ee08f628b300ba84655f10dc7f62a711bf7713d7ccdafb46d1dc9d7155d51"
Sep 16 18:45:17.599634 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:17.599616    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-k8kfm"
Sep 16 18:45:18.438355 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:18.438219    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:45:18.438465 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:18.438282    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:45:18.438530 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:18.438456    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pmm6" podUID="9f2230c8-6009-4227-8dbc-12e10ebe55f2"
Sep 16 18:45:18.438530 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:18.438498    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zp95h" podUID="d59dba60-7c0d-4e12-b53a-3462d4b9791a"
Sep 16 18:45:18.602924 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:18.602854    2554 generic.go:356] "Generic (PLEG): container finished" podID="698bb50e-290e-4842-af73-e8e8f7925f3a" containerID="f0ec01e2f416cfdb6021fadacb1532f63aa402f41e3349b99f3b8021c1a9e56f" exitCode=0
Sep 16 18:45:18.603254 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:18.602935    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frrd7" event={"ID":"698bb50e-290e-4842-af73-e8e8f7925f3a","Type":"ContainerDied","Data":"f0ec01e2f416cfdb6021fadacb1532f63aa402f41e3349b99f3b8021c1a9e56f"}
Sep 16 18:45:18.607010 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:18.606993    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovn-acl-logging/0.log"
Sep 16 18:45:18.607413 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:18.607388    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kdln4" event={"ID":"8f7856f8-39ee-46ec-8eb5-48dd0503e543","Type":"ContainerStarted","Data":"4b9d12981cecd587dfad683334388912a215a254716c4b791e0c9052bb46541e"}
Sep 16 18:45:18.607805 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:18.607600    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:45:18.607805 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:18.607674    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:45:18.607805 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:18.607742    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:45:18.623963 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:18.623939    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:45:18.624105 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:18.624089    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:45:18.629222 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:18.629203    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dvtnm"]
Sep 16 18:45:18.629309 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:18.629280    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:45:18.629365 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:18.629347    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvtnm" podUID="b17c052d-1029-47d7-960f-af161a00f71c"
Sep 16 18:45:18.634935 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:18.634796    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4pmm6"]
Sep 16 18:45:18.634935 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:18.634888    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:45:18.635099 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:18.634984    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pmm6" podUID="9f2230c8-6009-4227-8dbc-12e10ebe55f2"
Sep 16 18:45:18.635430 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:18.635413    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zp95h"]
Sep 16 18:45:18.635493 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:18.635475    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:45:18.635568 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:18.635553    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zp95h" podUID="d59dba60-7c0d-4e12-b53a-3462d4b9791a"
Sep 16 18:45:18.666172 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:18.666126    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kdln4" podStartSLOduration=8.829521924 podStartE2EDuration="27.666110612s" totalImagesPullingTime="18.836588688s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:18.664900538 +0000 UTC m=+28.740670169" watchObservedRunningTime="2026-09-16 18:45:18.666110612 +0000 UTC m=+28.741880242"
Sep 16 18:45:20.438771 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:20.438742    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:45:20.439178 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:20.438769    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:45:20.439178 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:20.438850    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvtnm" podUID="b17c052d-1029-47d7-960f-af161a00f71c"
Sep 16 18:45:20.439178 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:20.438940    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pmm6" podUID="9f2230c8-6009-4227-8dbc-12e10ebe55f2"
Sep 16 18:45:20.439178 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:20.438987    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:45:20.439178 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:20.439099    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zp95h" podUID="d59dba60-7c0d-4e12-b53a-3462d4b9791a"
Sep 16 18:45:20.613709 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:20.613638    2554 generic.go:356] "Generic (PLEG): container finished" podID="698bb50e-290e-4842-af73-e8e8f7925f3a" containerID="5256d3c148e8d72eb4de38b48e0449331c6162ffddf30ed61ceb48a78a6c0d64" exitCode=0
Sep 16 18:45:20.613870 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:20.613730    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frrd7" event={"ID":"698bb50e-290e-4842-af73-e8e8f7925f3a","Type":"ContainerDied","Data":"5256d3c148e8d72eb4de38b48e0449331c6162ffddf30ed61ceb48a78a6c0d64"}
Sep 16 18:45:22.438876 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:22.438660    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:45:22.439322 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:22.438654    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:45:22.439322 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:22.438974    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pmm6" podUID="9f2230c8-6009-4227-8dbc-12e10ebe55f2"
Sep 16 18:45:22.439322 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:22.439079    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zp95h" podUID="d59dba60-7c0d-4e12-b53a-3462d4b9791a"
Sep 16 18:45:22.439322 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:22.438669    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:45:22.439322 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:22.439170    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvtnm" podUID="b17c052d-1029-47d7-960f-af161a00f71c"
Sep 16 18:45:24.151788 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.151759    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f2230c8-6009-4227-8dbc-12e10ebe55f2-original-pull-secret\") pod \"global-pull-secret-syncer-4pmm6\" (UID: \"9f2230c8-6009-4227-8dbc-12e10ebe55f2\") " pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:45:24.152219 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.151843    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59dba60-7c0d-4e12-b53a-3462d4b9791a-metrics-certs\") pod \"network-metrics-daemon-zp95h\" (UID: \"d59dba60-7c0d-4e12-b53a-3462d4b9791a\") " pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:45:24.152219 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:24.151921    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:24.152219 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:24.151945    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:24.152219 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:24.151994    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f2230c8-6009-4227-8dbc-12e10ebe55f2-original-pull-secret podName:9f2230c8-6009-4227-8dbc-12e10ebe55f2 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:56.151975219 +0000 UTC m=+66.227744833 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9f2230c8-6009-4227-8dbc-12e10ebe55f2-original-pull-secret") pod "global-pull-secret-syncer-4pmm6" (UID: "9f2230c8-6009-4227-8dbc-12e10ebe55f2") : object "kube-system"/"original-pull-secret" not registered
Sep 16 18:45:24.152219 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:24.152017    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d59dba60-7c0d-4e12-b53a-3462d4b9791a-metrics-certs podName:d59dba60-7c0d-4e12-b53a-3462d4b9791a nodeName:}" failed. No retries permitted until 2026-09-16 18:45:56.152006787 +0000 UTC m=+66.227776396 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d59dba60-7c0d-4e12-b53a-3462d4b9791a-metrics-certs") pod "network-metrics-daemon-zp95h" (UID: "d59dba60-7c0d-4e12-b53a-3462d4b9791a") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 16 18:45:24.253105 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.253076    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jn69\" (UniqueName: \"kubernetes.io/projected/b17c052d-1029-47d7-960f-af161a00f71c-kube-api-access-2jn69\") pod \"network-check-target-dvtnm\" (UID: \"b17c052d-1029-47d7-960f-af161a00f71c\") " pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:45:24.253251 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:24.253210    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 16 18:45:24.253251 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:24.253223    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 16 18:45:24.253251 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:24.253246    2554 projected.go:196] Error preparing data for projected volume kube-api-access-2jn69 for pod openshift-network-diagnostics/network-check-target-dvtnm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:24.253361 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:24.253306    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b17c052d-1029-47d7-960f-af161a00f71c-kube-api-access-2jn69 podName:b17c052d-1029-47d7-960f-af161a00f71c nodeName:}" failed. No retries permitted until 2026-09-16 18:45:56.253287834 +0000 UTC m=+66.329057478 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2jn69" (UniqueName: "kubernetes.io/projected/b17c052d-1029-47d7-960f-af161a00f71c-kube-api-access-2jn69") pod "network-check-target-dvtnm" (UID: "b17c052d-1029-47d7-960f-af161a00f71c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 16 18:45:24.437861 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.437803    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:45:24.437861 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.437837    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:45:24.438044 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.437918    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:45:24.438044 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:24.437961    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zp95h" podUID="d59dba60-7c0d-4e12-b53a-3462d4b9791a"
Sep 16 18:45:24.438150 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:24.438080    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dvtnm" podUID="b17c052d-1029-47d7-960f-af161a00f71c"
Sep 16 18:45:24.438210 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:24.438165    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4pmm6" podUID="9f2230c8-6009-4227-8dbc-12e10ebe55f2"
Sep 16 18:45:24.626589 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.626556    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-241.ec2.internal" event="NodeReady"
Sep 16 18:45:24.626832 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.626675    2554 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 16 18:45:24.678853 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.678822    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tjwj4"]
Sep 16 18:45:24.699098 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.698983    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pbdzg"]
Sep 16 18:45:24.699240 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.699136    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tjwj4"
Sep 16 18:45:24.701060 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.701040    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 16 18:45:24.701308 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.701280    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 16 18:45:24.701544 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.701526    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6vj57\""
Sep 16 18:45:24.711283 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.711264    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tjwj4"]
Sep 16 18:45:24.711386 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.711303    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pbdzg"]
Sep 16 18:45:24.711386 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.711379    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pbdzg"
Sep 16 18:45:24.713931 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.713911    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 16 18:45:24.714216 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.714200    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 16 18:45:24.714301 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.714230    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 16 18:45:24.714301 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.714264    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2bmd5\""
Sep 16 18:45:24.857091 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.857056    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bg66\" (UniqueName: \"kubernetes.io/projected/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-kube-api-access-4bg66\") pod \"dns-default-tjwj4\" (UID: \"d0bc4f19-10ce-472d-b31b-dd58bb9d7afa\") " pod="openshift-dns/dns-default-tjwj4"
Sep 16 18:45:24.857330 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.857095    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6qbp\" (UniqueName: \"kubernetes.io/projected/d7440848-ec21-400a-9143-2b6794067e8a-kube-api-access-g6qbp\") pod \"ingress-canary-pbdzg\" (UID: \"d7440848-ec21-400a-9143-2b6794067e8a\") " pod="openshift-ingress-canary/ingress-canary-pbdzg"
Sep 16 18:45:24.857330 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.857155    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-config-volume\") pod \"dns-default-tjwj4\" (UID: \"d0bc4f19-10ce-472d-b31b-dd58bb9d7afa\") " pod="openshift-dns/dns-default-tjwj4"
Sep 16 18:45:24.857330 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.857247    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-metrics-tls\") pod \"dns-default-tjwj4\" (UID: \"d0bc4f19-10ce-472d-b31b-dd58bb9d7afa\") " pod="openshift-dns/dns-default-tjwj4"
Sep 16 18:45:24.857330 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.857286    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-tmp-dir\") pod \"dns-default-tjwj4\" (UID: \"d0bc4f19-10ce-472d-b31b-dd58bb9d7afa\") " pod="openshift-dns/dns-default-tjwj4"
Sep 16 18:45:24.857491 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.857340    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7440848-ec21-400a-9143-2b6794067e8a-cert\") pod \"ingress-canary-pbdzg\" (UID: \"d7440848-ec21-400a-9143-2b6794067e8a\") " pod="openshift-ingress-canary/ingress-canary-pbdzg"
Sep 16 18:45:24.958604 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.958532    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-tmp-dir\") pod \"dns-default-tjwj4\" (UID: \"d0bc4f19-10ce-472d-b31b-dd58bb9d7afa\") " pod="openshift-dns/dns-default-tjwj4"
Sep 16 18:45:24.958604 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.958577    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7440848-ec21-400a-9143-2b6794067e8a-cert\") pod \"ingress-canary-pbdzg\" (UID: \"d7440848-ec21-400a-9143-2b6794067e8a\") " pod="openshift-ingress-canary/ingress-canary-pbdzg"
Sep 16 18:45:24.958870 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.958625    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bg66\" (UniqueName: \"kubernetes.io/projected/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-kube-api-access-4bg66\") pod \"dns-default-tjwj4\" (UID: \"d0bc4f19-10ce-472d-b31b-dd58bb9d7afa\") " pod="openshift-dns/dns-default-tjwj4"
Sep 16 18:45:24.958870 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.958656    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6qbp\" (UniqueName: \"kubernetes.io/projected/d7440848-ec21-400a-9143-2b6794067e8a-kube-api-access-g6qbp\") pod \"ingress-canary-pbdzg\" (UID: \"d7440848-ec21-400a-9143-2b6794067e8a\") " pod="openshift-ingress-canary/ingress-canary-pbdzg"
Sep 16 18:45:24.958870 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.958701    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-config-volume\") pod \"dns-default-tjwj4\" (UID: \"d0bc4f19-10ce-472d-b31b-dd58bb9d7afa\") " pod="openshift-dns/dns-default-tjwj4"
Sep 16 18:45:24.958870 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.958750    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-metrics-tls\") pod \"dns-default-tjwj4\" (UID: \"d0bc4f19-10ce-472d-b31b-dd58bb9d7afa\") " pod="openshift-dns/dns-default-tjwj4"
Sep 16 18:45:24.958870 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:24.958845    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:24.959044 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:24.958889    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-metrics-tls podName:d0bc4f19-10ce-472d-b31b-dd58bb9d7afa nodeName:}" failed. No retries permitted until 2026-09-16 18:45:25.458876194 +0000 UTC m=+35.534645804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-metrics-tls") pod "dns-default-tjwj4" (UID: "d0bc4f19-10ce-472d-b31b-dd58bb9d7afa") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:24.959044 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.958923    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-tmp-dir\") pod \"dns-default-tjwj4\" (UID: \"d0bc4f19-10ce-472d-b31b-dd58bb9d7afa\") " pod="openshift-dns/dns-default-tjwj4"
Sep 16 18:45:24.959635 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.959559    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-config-volume\") pod \"dns-default-tjwj4\" (UID: \"d0bc4f19-10ce-472d-b31b-dd58bb9d7afa\") " pod="openshift-dns/dns-default-tjwj4"
Sep 16 18:45:24.963832 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.963806    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d7440848-ec21-400a-9143-2b6794067e8a-cert\") pod \"ingress-canary-pbdzg\" (UID: \"d7440848-ec21-400a-9143-2b6794067e8a\") " pod="openshift-ingress-canary/ingress-canary-pbdzg"
Sep 16 18:45:24.967803 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.967753    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bg66\" (UniqueName: \"kubernetes.io/projected/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-kube-api-access-4bg66\") pod \"dns-default-tjwj4\" (UID: \"d0bc4f19-10ce-472d-b31b-dd58bb9d7afa\") " pod="openshift-dns/dns-default-tjwj4"
Sep 16 18:45:24.968006 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:24.967969    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6qbp\" (UniqueName: \"kubernetes.io/projected/d7440848-ec21-400a-9143-2b6794067e8a-kube-api-access-g6qbp\") pod \"ingress-canary-pbdzg\" (UID: \"d7440848-ec21-400a-9143-2b6794067e8a\") " pod="openshift-ingress-canary/ingress-canary-pbdzg"
Sep 16 18:45:25.020877 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:25.020856    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pbdzg"
Sep 16 18:45:25.351910 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:25.351886    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pbdzg"]
Sep 16 18:45:25.353972 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:45:25.353949    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7440848_ec21_400a_9143_2b6794067e8a.slice/crio-31bba1c6e87889d9ac8f8ad7df40bbd25e13b9fab3dee20d116647750b5e1a0a WatchSource:0}: Error finding container 31bba1c6e87889d9ac8f8ad7df40bbd25e13b9fab3dee20d116647750b5e1a0a: Status 404 returned error can't find the container with id 31bba1c6e87889d9ac8f8ad7df40bbd25e13b9fab3dee20d116647750b5e1a0a
Sep 16 18:45:25.462972 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:25.462864    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-metrics-tls\") pod \"dns-default-tjwj4\" (UID: \"d0bc4f19-10ce-472d-b31b-dd58bb9d7afa\") " pod="openshift-dns/dns-default-tjwj4"
Sep 16 18:45:25.463076 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:25.463016    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:25.463076 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:25.463075    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-metrics-tls podName:d0bc4f19-10ce-472d-b31b-dd58bb9d7afa nodeName:}" failed. No retries permitted until 2026-09-16 18:45:26.463060876 +0000 UTC m=+36.538830486 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-metrics-tls") pod "dns-default-tjwj4" (UID: "d0bc4f19-10ce-472d-b31b-dd58bb9d7afa") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:25.628161 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:25.628138    2554 generic.go:356] "Generic (PLEG): container finished" podID="698bb50e-290e-4842-af73-e8e8f7925f3a" containerID="5d319674e7f585e2dcf209bc4bea66c951470196b5b55e6ec0c71c0e5a731308" exitCode=0
Sep 16 18:45:25.628277 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:25.628223    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frrd7" event={"ID":"698bb50e-290e-4842-af73-e8e8f7925f3a","Type":"ContainerDied","Data":"5d319674e7f585e2dcf209bc4bea66c951470196b5b55e6ec0c71c0e5a731308"}
Sep 16 18:45:25.629373 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:25.629354    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pbdzg" event={"ID":"d7440848-ec21-400a-9143-2b6794067e8a","Type":"ContainerStarted","Data":"31bba1c6e87889d9ac8f8ad7df40bbd25e13b9fab3dee20d116647750b5e1a0a"}
Sep 16 18:45:26.437934 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:26.437902    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:45:26.438406 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:26.437902    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:45:26.438406 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:26.437902    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:45:26.440969 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:26.440783    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:45:26.440969 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:26.440810    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:45:26.440969 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:26.440829    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rlx6q\""
Sep 16 18:45:26.440969 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:26.440838    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2qmgm\""
Sep 16 18:45:26.440969 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:26.440866    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:45:26.440969 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:26.440789    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:45:26.470761 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:26.470694    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-metrics-tls\") pod \"dns-default-tjwj4\" (UID: \"d0bc4f19-10ce-472d-b31b-dd58bb9d7afa\") " pod="openshift-dns/dns-default-tjwj4"
Sep 16 18:45:26.470862 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:26.470845    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:26.471366 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:26.471349    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-metrics-tls podName:d0bc4f19-10ce-472d-b31b-dd58bb9d7afa nodeName:}" failed. No retries permitted until 2026-09-16 18:45:28.471327007 +0000 UTC m=+38.547096631 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-metrics-tls") pod "dns-default-tjwj4" (UID: "d0bc4f19-10ce-472d-b31b-dd58bb9d7afa") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:26.635614 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:26.635584    2554 generic.go:356] "Generic (PLEG): container finished" podID="698bb50e-290e-4842-af73-e8e8f7925f3a" containerID="1314fee8e003f2cda64a7549a2e3d6453ed46c749bec3bd5e2968d6d0e8d4c4e" exitCode=0
Sep 16 18:45:26.635762 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:26.635617    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frrd7" event={"ID":"698bb50e-290e-4842-af73-e8e8f7925f3a","Type":"ContainerDied","Data":"1314fee8e003f2cda64a7549a2e3d6453ed46c749bec3bd5e2968d6d0e8d4c4e"}
Sep 16 18:45:27.641304 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:27.641274    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-frrd7" event={"ID":"698bb50e-290e-4842-af73-e8e8f7925f3a","Type":"ContainerStarted","Data":"b48849585f8fa61932e9bd46eefa3af59a45294e9cced97c7f33c491b1515cf8"}
Sep 16 18:45:27.680298 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:27.680243    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-frrd7" podStartSLOduration=7.423259097 podStartE2EDuration="36.680221738s" totalImagesPullingTime="29.256962641s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:27.676393382 +0000 UTC m=+37.752163013" watchObservedRunningTime="2026-09-16 18:45:27.680221738 +0000 UTC m=+37.755991373"
Sep 16 18:45:28.486014 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:28.485944    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-metrics-tls\") pod \"dns-default-tjwj4\" (UID: \"d0bc4f19-10ce-472d-b31b-dd58bb9d7afa\") " pod="openshift-dns/dns-default-tjwj4"
Sep 16 18:45:28.486139 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:28.486089    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:28.486178 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:28.486143    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-metrics-tls podName:d0bc4f19-10ce-472d-b31b-dd58bb9d7afa nodeName:}" failed. No retries permitted until 2026-09-16 18:45:32.486129926 +0000 UTC m=+42.561899539 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-metrics-tls") pod "dns-default-tjwj4" (UID: "d0bc4f19-10ce-472d-b31b-dd58bb9d7afa") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:28.645463 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:28.645301    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pbdzg" event={"ID":"d7440848-ec21-400a-9143-2b6794067e8a","Type":"ContainerStarted","Data":"8730706be29bbe1bf336531eff1d21b8a2fb96285049907a5e9bb50055d17735"}
Sep 16 18:45:32.512882 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:32.512847    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-metrics-tls\") pod \"dns-default-tjwj4\" (UID: \"d0bc4f19-10ce-472d-b31b-dd58bb9d7afa\") " pod="openshift-dns/dns-default-tjwj4"
Sep 16 18:45:32.513222 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:32.512988    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:32.513222 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:32.513046    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-metrics-tls podName:d0bc4f19-10ce-472d-b31b-dd58bb9d7afa nodeName:}" failed. No retries permitted until 2026-09-16 18:45:40.513029726 +0000 UTC m=+50.588799359 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-metrics-tls") pod "dns-default-tjwj4" (UID: "d0bc4f19-10ce-472d-b31b-dd58bb9d7afa") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:40.561322 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:40.561288    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-metrics-tls\") pod \"dns-default-tjwj4\" (UID: \"d0bc4f19-10ce-472d-b31b-dd58bb9d7afa\") " pod="openshift-dns/dns-default-tjwj4"
Sep 16 18:45:40.561707 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:40.561398    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:40.561707 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:40.561444    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-metrics-tls podName:d0bc4f19-10ce-472d-b31b-dd58bb9d7afa nodeName:}" failed. No retries permitted until 2026-09-16 18:45:56.561430755 +0000 UTC m=+66.637200397 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-metrics-tls") pod "dns-default-tjwj4" (UID: "d0bc4f19-10ce-472d-b31b-dd58bb9d7afa") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:43.190557 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.190493    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pbdzg" podStartSLOduration=16.674761987 podStartE2EDuration="19.190469516s" totalImagesPullingTime="2.515707529s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:28.662584147 +0000 UTC m=+38.738353780" watchObservedRunningTime="2026-09-16 18:45:43.190469516 +0000 UTC m=+53.266239151"
Sep 16 18:45:43.191106 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.191003    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-llwg5"]
Sep 16 18:45:43.192843 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.192819    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-llwg5"
Sep 16 18:45:43.195930 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.195903    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:45:43.195930 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.195921    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 16 18:45:43.196105 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.195909    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 16 18:45:43.196105 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.195910    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-bfwd7\""
Sep 16 18:45:43.196362 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.196346    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 16 18:45:43.203461 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.203436    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 16 18:45:43.205517 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.205492    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-llwg5"]
Sep 16 18:45:43.277545 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.277506    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3589130e-56b7-4ce2-ad13-55804eabd1b3-config\") pod \"console-operator-696884fb9c-llwg5\" (UID: \"3589130e-56b7-4ce2-ad13-55804eabd1b3\") " pod="openshift-console-operator/console-operator-696884fb9c-llwg5"
Sep 16 18:45:43.277675 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.277560    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3589130e-56b7-4ce2-ad13-55804eabd1b3-serving-cert\") pod \"console-operator-696884fb9c-llwg5\" (UID: \"3589130e-56b7-4ce2-ad13-55804eabd1b3\") " pod="openshift-console-operator/console-operator-696884fb9c-llwg5"
Sep 16 18:45:43.277675 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.277590    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2l6l\" (UniqueName: \"kubernetes.io/projected/3589130e-56b7-4ce2-ad13-55804eabd1b3-kube-api-access-f2l6l\") pod \"console-operator-696884fb9c-llwg5\" (UID: \"3589130e-56b7-4ce2-ad13-55804eabd1b3\") " pod="openshift-console-operator/console-operator-696884fb9c-llwg5"
Sep 16 18:45:43.277675 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.277639    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3589130e-56b7-4ce2-ad13-55804eabd1b3-trusted-ca\") pod \"console-operator-696884fb9c-llwg5\" (UID: \"3589130e-56b7-4ce2-ad13-55804eabd1b3\") " pod="openshift-console-operator/console-operator-696884fb9c-llwg5"
Sep 16 18:45:43.378032 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.378004    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3589130e-56b7-4ce2-ad13-55804eabd1b3-config\") pod \"console-operator-696884fb9c-llwg5\" (UID: \"3589130e-56b7-4ce2-ad13-55804eabd1b3\") " pod="openshift-console-operator/console-operator-696884fb9c-llwg5"
Sep 16 18:45:43.378136 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.378049    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3589130e-56b7-4ce2-ad13-55804eabd1b3-serving-cert\") pod \"console-operator-696884fb9c-llwg5\" (UID: \"3589130e-56b7-4ce2-ad13-55804eabd1b3\") " pod="openshift-console-operator/console-operator-696884fb9c-llwg5"
Sep 16 18:45:43.378136 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.378074    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2l6l\" (UniqueName: \"kubernetes.io/projected/3589130e-56b7-4ce2-ad13-55804eabd1b3-kube-api-access-f2l6l\") pod \"console-operator-696884fb9c-llwg5\" (UID: \"3589130e-56b7-4ce2-ad13-55804eabd1b3\") " pod="openshift-console-operator/console-operator-696884fb9c-llwg5"
Sep 16 18:45:43.378136 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.378101    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3589130e-56b7-4ce2-ad13-55804eabd1b3-trusted-ca\") pod \"console-operator-696884fb9c-llwg5\" (UID: \"3589130e-56b7-4ce2-ad13-55804eabd1b3\") " pod="openshift-console-operator/console-operator-696884fb9c-llwg5"
Sep 16 18:45:43.378851 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.378825    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3589130e-56b7-4ce2-ad13-55804eabd1b3-config\") pod \"console-operator-696884fb9c-llwg5\" (UID: \"3589130e-56b7-4ce2-ad13-55804eabd1b3\") " pod="openshift-console-operator/console-operator-696884fb9c-llwg5"
Sep 16 18:45:43.379042 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.379018    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3589130e-56b7-4ce2-ad13-55804eabd1b3-trusted-ca\") pod \"console-operator-696884fb9c-llwg5\" (UID: \"3589130e-56b7-4ce2-ad13-55804eabd1b3\") " pod="openshift-console-operator/console-operator-696884fb9c-llwg5"
Sep 16 18:45:43.381999 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.381972    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3589130e-56b7-4ce2-ad13-55804eabd1b3-serving-cert\") pod \"console-operator-696884fb9c-llwg5\" (UID: \"3589130e-56b7-4ce2-ad13-55804eabd1b3\") " pod="openshift-console-operator/console-operator-696884fb9c-llwg5"
Sep 16 18:45:43.387161 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.387139    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2l6l\" (UniqueName: \"kubernetes.io/projected/3589130e-56b7-4ce2-ad13-55804eabd1b3-kube-api-access-f2l6l\") pod \"console-operator-696884fb9c-llwg5\" (UID: \"3589130e-56b7-4ce2-ad13-55804eabd1b3\") " pod="openshift-console-operator/console-operator-696884fb9c-llwg5"
Sep 16 18:45:43.399677 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.399656    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-ln8nz"]
Sep 16 18:45:43.401444 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.401430    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ln8nz"
Sep 16 18:45:43.403761 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.403689    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 16 18:45:43.404232 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.404208    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 16 18:45:43.404232 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.404226    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 16 18:45:43.404364 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.404229    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-w9rhs\""
Sep 16 18:45:43.406536 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.406516    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 16 18:45:43.422087 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.422067    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-ln8nz"]
Sep 16 18:45:43.478771 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.478710    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-ln8nz\" (UID: \"b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ln8nz"
Sep 16 18:45:43.478771 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.478749    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-ln8nz\" (UID: \"b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ln8nz"
Sep 16 18:45:43.478879 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.478808    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p652\" (UniqueName: \"kubernetes.io/projected/b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5-kube-api-access-6p652\") pod \"cluster-monitoring-operator-6cd954b897-ln8nz\" (UID: \"b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ln8nz"
Sep 16 18:45:43.502825 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.502804    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-696884fb9c-llwg5"
Sep 16 18:45:43.515226 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.515205    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-5qjn2"]
Sep 16 18:45:43.517276 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.517259    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5qjn2"
Sep 16 18:45:43.519586 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.519567    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 16 18:45:43.520101 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.520081    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 16 18:45:43.520173 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.520149    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 16 18:45:43.520173 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.520163    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5znwp\""
Sep 16 18:45:43.520653 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.520638    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 16 18:45:43.531026 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.531008    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-5qjn2"]
Sep 16 18:45:43.579500 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.579477    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p652\" (UniqueName: \"kubernetes.io/projected/b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5-kube-api-access-6p652\") pod \"cluster-monitoring-operator-6cd954b897-ln8nz\" (UID: \"b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ln8nz"
Sep 16 18:45:43.579624 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.579515    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/902005e5-c11d-408c-a3f6-82482c7b4e45-serving-cert\") pod \"service-ca-operator-cf7c776b7-5qjn2\" (UID: \"902005e5-c11d-408c-a3f6-82482c7b4e45\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5qjn2"
Sep 16 18:45:43.579624 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.579574    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/902005e5-c11d-408c-a3f6-82482c7b4e45-config\") pod \"service-ca-operator-cf7c776b7-5qjn2\" (UID: \"902005e5-c11d-408c-a3f6-82482c7b4e45\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5qjn2"
Sep 16 18:45:43.579624 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.579602    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbk42\" (UniqueName: \"kubernetes.io/projected/902005e5-c11d-408c-a3f6-82482c7b4e45-kube-api-access-sbk42\") pod \"service-ca-operator-cf7c776b7-5qjn2\" (UID: \"902005e5-c11d-408c-a3f6-82482c7b4e45\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5qjn2"
Sep 16 18:45:43.579806 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.579667    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-ln8nz\" (UID: \"b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ln8nz"
Sep 16 18:45:43.579806 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.579693    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-ln8nz\" (UID: \"b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ln8nz"
Sep 16 18:45:43.579910 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:43.579831    2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:45:43.579910 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:43.579894    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5-cluster-monitoring-operator-tls podName:b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:44.079874653 +0000 UTC m=+54.155644286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-ln8nz" (UID: "b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:45:43.580559 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.580539    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5-telemetry-config\") pod \"cluster-monitoring-operator-6cd954b897-ln8nz\" (UID: \"b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ln8nz"
Sep 16 18:45:43.592702 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.592682    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p652\" (UniqueName: \"kubernetes.io/projected/b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5-kube-api-access-6p652\") pod \"cluster-monitoring-operator-6cd954b897-ln8nz\" (UID: \"b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ln8nz"
Sep 16 18:45:43.643206 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.643184    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-558956f4c-ptzxv"]
Sep 16 18:45:43.644890 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.644876    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:43.647478 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.647456    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 16 18:45:43.647805 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.647787    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hvbtm\""
Sep 16 18:45:43.647911 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.647895    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 16 18:45:43.647993 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.647979    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 16 18:45:43.658055 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.658036    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 16 18:45:43.663581 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.663561    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-558956f4c-ptzxv"]
Sep 16 18:45:43.680414 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.680396    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2688504b-6527-42f6-a904-8ef74b146bc7-trusted-ca\") pod \"image-registry-558956f4c-ptzxv\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") " pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:43.680491 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.680447    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/902005e5-c11d-408c-a3f6-82482c7b4e45-serving-cert\") pod \"service-ca-operator-cf7c776b7-5qjn2\" (UID: \"902005e5-c11d-408c-a3f6-82482c7b4e45\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5qjn2"
Sep 16 18:45:43.680491 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.680480    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-registry-tls\") pod \"image-registry-558956f4c-ptzxv\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") " pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:43.680569 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.680499    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2688504b-6527-42f6-a904-8ef74b146bc7-registry-certificates\") pod \"image-registry-558956f4c-ptzxv\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") " pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:43.680569 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.680521    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/902005e5-c11d-408c-a3f6-82482c7b4e45-config\") pod \"service-ca-operator-cf7c776b7-5qjn2\" (UID: \"902005e5-c11d-408c-a3f6-82482c7b4e45\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5qjn2"
Sep 16 18:45:43.680569 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.680539    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbk42\" (UniqueName: \"kubernetes.io/projected/902005e5-c11d-408c-a3f6-82482c7b4e45-kube-api-access-sbk42\") pod \"service-ca-operator-cf7c776b7-5qjn2\" (UID: \"902005e5-c11d-408c-a3f6-82482c7b4e45\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5qjn2"
Sep 16 18:45:43.680569 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.680561    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2688504b-6527-42f6-a904-8ef74b146bc7-ca-trust-extracted\") pod \"image-registry-558956f4c-ptzxv\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") " pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:43.680744 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.680584    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-bound-sa-token\") pod \"image-registry-558956f4c-ptzxv\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") " pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:43.680744 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.680636    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mp7j\" (UniqueName: \"kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-kube-api-access-8mp7j\") pod \"image-registry-558956f4c-ptzxv\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") " pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:43.680744 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.680695    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2688504b-6527-42f6-a904-8ef74b146bc7-image-registry-private-configuration\") pod \"image-registry-558956f4c-ptzxv\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") " pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:43.680849 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.680769    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2688504b-6527-42f6-a904-8ef74b146bc7-installation-pull-secrets\") pod \"image-registry-558956f4c-ptzxv\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") " pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:43.681020 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.681004    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/902005e5-c11d-408c-a3f6-82482c7b4e45-config\") pod \"service-ca-operator-cf7c776b7-5qjn2\" (UID: \"902005e5-c11d-408c-a3f6-82482c7b4e45\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5qjn2"
Sep 16 18:45:43.682501 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.682485    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/902005e5-c11d-408c-a3f6-82482c7b4e45-serving-cert\") pod \"service-ca-operator-cf7c776b7-5qjn2\" (UID: \"902005e5-c11d-408c-a3f6-82482c7b4e45\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5qjn2"
Sep 16 18:45:43.688250 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.688233    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbk42\" (UniqueName: \"kubernetes.io/projected/902005e5-c11d-408c-a3f6-82482c7b4e45-kube-api-access-sbk42\") pod \"service-ca-operator-cf7c776b7-5qjn2\" (UID: \"902005e5-c11d-408c-a3f6-82482c7b4e45\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5qjn2"
Sep 16 18:45:43.770599 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.770579    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-696884fb9c-llwg5"]
Sep 16 18:45:43.773306 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:45:43.773282    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3589130e_56b7_4ce2_ad13_55804eabd1b3.slice/crio-cf73cccd4bbda0578080bc4ea712b175eafed2d2227b790f7754969c9ca27a2b WatchSource:0}: Error finding container cf73cccd4bbda0578080bc4ea712b175eafed2d2227b790f7754969c9ca27a2b: Status 404 returned error can't find the container with id cf73cccd4bbda0578080bc4ea712b175eafed2d2227b790f7754969c9ca27a2b
Sep 16 18:45:43.781985 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.781965    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-registry-tls\") pod \"image-registry-558956f4c-ptzxv\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") " pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:43.782054 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.781998    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2688504b-6527-42f6-a904-8ef74b146bc7-registry-certificates\") pod \"image-registry-558956f4c-ptzxv\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") " pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:43.782054 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.782028    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2688504b-6527-42f6-a904-8ef74b146bc7-ca-trust-extracted\") pod \"image-registry-558956f4c-ptzxv\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") " pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:43.782132 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:43.782106    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:43.782132 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:43.782122    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-558956f4c-ptzxv: secret "image-registry-tls" not found
Sep 16 18:45:43.782288 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:43.782167    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-registry-tls podName:2688504b-6527-42f6-a904-8ef74b146bc7 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:44.282151175 +0000 UTC m=+54.357920785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-registry-tls") pod "image-registry-558956f4c-ptzxv" (UID: "2688504b-6527-42f6-a904-8ef74b146bc7") : secret "image-registry-tls" not found
Sep 16 18:45:43.782288 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.782192    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-bound-sa-token\") pod \"image-registry-558956f4c-ptzxv\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") " pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:43.782288 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.782248    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mp7j\" (UniqueName: \"kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-kube-api-access-8mp7j\") pod \"image-registry-558956f4c-ptzxv\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") " pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:43.782288 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.782282    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2688504b-6527-42f6-a904-8ef74b146bc7-image-registry-private-configuration\") pod \"image-registry-558956f4c-ptzxv\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") " pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:43.782561 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.782310    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2688504b-6527-42f6-a904-8ef74b146bc7-installation-pull-secrets\") pod \"image-registry-558956f4c-ptzxv\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") " pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:43.782561 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.782345    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2688504b-6527-42f6-a904-8ef74b146bc7-ca-trust-extracted\") pod \"image-registry-558956f4c-ptzxv\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") " pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:43.782561 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.782373    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2688504b-6527-42f6-a904-8ef74b146bc7-trusted-ca\") pod \"image-registry-558956f4c-ptzxv\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") " pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:43.782739 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.782576    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2688504b-6527-42f6-a904-8ef74b146bc7-registry-certificates\") pod \"image-registry-558956f4c-ptzxv\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") " pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:43.783119 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.783099    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2688504b-6527-42f6-a904-8ef74b146bc7-trusted-ca\") pod \"image-registry-558956f4c-ptzxv\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") " pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:43.784438 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.784415    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2688504b-6527-42f6-a904-8ef74b146bc7-image-registry-private-configuration\") pod \"image-registry-558956f4c-ptzxv\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") " pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:43.784480 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.784453    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2688504b-6527-42f6-a904-8ef74b146bc7-installation-pull-secrets\") pod \"image-registry-558956f4c-ptzxv\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") " pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:43.789923 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.789907    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mp7j\" (UniqueName: \"kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-kube-api-access-8mp7j\") pod \"image-registry-558956f4c-ptzxv\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") " pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:43.790214 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.790198    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-bound-sa-token\") pod \"image-registry-558956f4c-ptzxv\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") " pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:43.825671 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:43.825654    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5qjn2"
Sep 16 18:45:44.085296 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:44.085234    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-ln8nz\" (UID: \"b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ln8nz"
Sep 16 18:45:44.085484 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:44.085381    2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:45:44.085484 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:44.085451    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5-cluster-monitoring-operator-tls podName:b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:45.085432567 +0000 UTC m=+55.161202176 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-ln8nz" (UID: "b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:45:44.096982 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:44.096955    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-5qjn2"]
Sep 16 18:45:44.101556 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:45:44.101530    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod902005e5_c11d_408c_a3f6_82482c7b4e45.slice/crio-af66bd201e6bdebc281febdc56d8c81bf76409f812b57c86f6c89c5acc2698ab WatchSource:0}: Error finding container af66bd201e6bdebc281febdc56d8c81bf76409f812b57c86f6c89c5acc2698ab: Status 404 returned error can't find the container with id af66bd201e6bdebc281febdc56d8c81bf76409f812b57c86f6c89c5acc2698ab
Sep 16 18:45:44.286941 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:44.286918    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-registry-tls\") pod \"image-registry-558956f4c-ptzxv\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") " pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:44.287242 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:44.287025    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:44.287242 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:44.287043    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-558956f4c-ptzxv: secret "image-registry-tls" not found
Sep 16 18:45:44.287242 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:44.287095    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-registry-tls podName:2688504b-6527-42f6-a904-8ef74b146bc7 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:45.287082715 +0000 UTC m=+55.362852325 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-registry-tls") pod "image-registry-558956f4c-ptzxv" (UID: "2688504b-6527-42f6-a904-8ef74b146bc7") : secret "image-registry-tls" not found
Sep 16 18:45:44.683313 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:44.683275    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5qjn2" event={"ID":"902005e5-c11d-408c-a3f6-82482c7b4e45","Type":"ContainerStarted","Data":"af66bd201e6bdebc281febdc56d8c81bf76409f812b57c86f6c89c5acc2698ab"}
Sep 16 18:45:44.684488 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:44.684458    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-llwg5" event={"ID":"3589130e-56b7-4ce2-ad13-55804eabd1b3","Type":"ContainerStarted","Data":"cf73cccd4bbda0578080bc4ea712b175eafed2d2227b790f7754969c9ca27a2b"}
Sep 16 18:45:45.093047 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:45.093012    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-ln8nz\" (UID: \"b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ln8nz"
Sep 16 18:45:45.093228 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:45.093130    2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:45:45.093228 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:45.093206    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5-cluster-monitoring-operator-tls podName:b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:47.093186651 +0000 UTC m=+57.168956281 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-ln8nz" (UID: "b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:45:45.295113 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:45.295071    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-registry-tls\") pod \"image-registry-558956f4c-ptzxv\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") " pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:45.295561 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:45.295255    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:45.295561 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:45.295282    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-558956f4c-ptzxv: secret "image-registry-tls" not found
Sep 16 18:45:45.295561 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:45.295360    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-registry-tls podName:2688504b-6527-42f6-a904-8ef74b146bc7 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:47.295336101 +0000 UTC m=+57.371105717 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-registry-tls") pod "image-registry-558956f4c-ptzxv" (UID: "2688504b-6527-42f6-a904-8ef74b146bc7") : secret "image-registry-tls" not found
Sep 16 18:45:46.692711 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:46.692668    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5qjn2" event={"ID":"902005e5-c11d-408c-a3f6-82482c7b4e45","Type":"ContainerStarted","Data":"6a6632232aaf33cec28404734832bb646109cbeb31bc226a061530f790000733"}
Sep 16 18:45:46.695157 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:46.695132    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/0.log"
Sep 16 18:45:46.695283 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:46.695168    2554 generic.go:356] "Generic (PLEG): container finished" podID="3589130e-56b7-4ce2-ad13-55804eabd1b3" containerID="9bf9bd2823bcda3b39ed34ad76e6b9be36f321b0b4f989cab850f4949930993b" exitCode=255
Sep 16 18:45:46.695283 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:46.695199    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-llwg5" event={"ID":"3589130e-56b7-4ce2-ad13-55804eabd1b3","Type":"ContainerDied","Data":"9bf9bd2823bcda3b39ed34ad76e6b9be36f321b0b4f989cab850f4949930993b"}
Sep 16 18:45:46.695500 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:46.695482    2554 scope.go:117] "RemoveContainer" containerID="9bf9bd2823bcda3b39ed34ad76e6b9be36f321b0b4f989cab850f4949930993b"
Sep 16 18:45:46.709897 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:46.708184    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5qjn2" podStartSLOduration=1.23933822 podStartE2EDuration="3.708165516s" totalImagesPullingTime="2.468827296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:46.707805144 +0000 UTC m=+56.783574778" watchObservedRunningTime="2026-09-16 18:45:46.708165516 +0000 UTC m=+56.783935150"
Sep 16 18:45:47.108908 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:47.108815    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-ln8nz\" (UID: \"b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ln8nz"
Sep 16 18:45:47.109055 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:47.109000    2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:45:47.109098 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:47.109087    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5-cluster-monitoring-operator-tls podName:b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:51.1090653 +0000 UTC m=+61.184834919 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-ln8nz" (UID: "b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:45:47.311200 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:47.311159    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-registry-tls\") pod \"image-registry-558956f4c-ptzxv\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") " pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:47.311346 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:47.311301    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:47.311346 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:47.311318    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-558956f4c-ptzxv: secret "image-registry-tls" not found
Sep 16 18:45:47.311409 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:47.311370    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-registry-tls podName:2688504b-6527-42f6-a904-8ef74b146bc7 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:51.311354792 +0000 UTC m=+61.387124416 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-registry-tls") pod "image-registry-558956f4c-ptzxv" (UID: "2688504b-6527-42f6-a904-8ef74b146bc7") : secret "image-registry-tls" not found
Sep 16 18:45:47.699387 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:47.699339    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/1.log"
Sep 16 18:45:47.699831 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:47.699780    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/0.log"
Sep 16 18:45:47.699831 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:47.699814    2554 generic.go:356] "Generic (PLEG): container finished" podID="3589130e-56b7-4ce2-ad13-55804eabd1b3" containerID="c9b969c4b66bedcd5da83befa495bbb975baee6be56ae81e982dd36cedddd6eb" exitCode=255
Sep 16 18:45:47.699935 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:47.699883    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-llwg5" event={"ID":"3589130e-56b7-4ce2-ad13-55804eabd1b3","Type":"ContainerDied","Data":"c9b969c4b66bedcd5da83befa495bbb975baee6be56ae81e982dd36cedddd6eb"}
Sep 16 18:45:47.699982 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:47.699947    2554 scope.go:117] "RemoveContainer" containerID="9bf9bd2823bcda3b39ed34ad76e6b9be36f321b0b4f989cab850f4949930993b"
Sep 16 18:45:47.700146 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:47.700119    2554 scope.go:117] "RemoveContainer" containerID="c9b969c4b66bedcd5da83befa495bbb975baee6be56ae81e982dd36cedddd6eb"
Sep 16 18:45:47.700341 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:47.700307    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-llwg5_openshift-console-operator(3589130e-56b7-4ce2-ad13-55804eabd1b3)\"" pod="openshift-console-operator/console-operator-696884fb9c-llwg5" podUID="3589130e-56b7-4ce2-ad13-55804eabd1b3"
Sep 16 18:45:48.574802 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.574765    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nrxcw"]
Sep 16 18:45:48.577419 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.577401    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nrxcw"
Sep 16 18:45:48.579372 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.579347    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 16 18:45:48.580075 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.580049    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 16 18:45:48.580212 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.580192    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 16 18:45:48.580683 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.580667    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sppxv\""
Sep 16 18:45:48.580938 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.580923    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 16 18:45:48.592632 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.592608    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nrxcw"]
Sep 16 18:45:48.621273 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.621244    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nrxcw\" (UID: \"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88\") " pod="openshift-insights/insights-runtime-extractor-nrxcw"
Sep 16 18:45:48.621402 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.621358    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-data-volume\") pod \"insights-runtime-extractor-nrxcw\" (UID: \"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88\") " pod="openshift-insights/insights-runtime-extractor-nrxcw"
Sep 16 18:45:48.621473 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.621453    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nrxcw\" (UID: \"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88\") " pod="openshift-insights/insights-runtime-extractor-nrxcw"
Sep 16 18:45:48.621546 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.621524    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtpf5\" (UniqueName: \"kubernetes.io/projected/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-kube-api-access-rtpf5\") pod \"insights-runtime-extractor-nrxcw\" (UID: \"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88\") " pod="openshift-insights/insights-runtime-extractor-nrxcw"
Sep 16 18:45:48.621630 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.621593    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-crio-socket\") pod \"insights-runtime-extractor-nrxcw\" (UID: \"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88\") " pod="openshift-insights/insights-runtime-extractor-nrxcw"
Sep 16 18:45:48.703471 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.703450    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/1.log"
Sep 16 18:45:48.703793 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.703705    2554 scope.go:117] "RemoveContainer" containerID="c9b969c4b66bedcd5da83befa495bbb975baee6be56ae81e982dd36cedddd6eb"
Sep 16 18:45:48.703879 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:48.703861    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-llwg5_openshift-console-operator(3589130e-56b7-4ce2-ad13-55804eabd1b3)\"" pod="openshift-console-operator/console-operator-696884fb9c-llwg5" podUID="3589130e-56b7-4ce2-ad13-55804eabd1b3"
Sep 16 18:45:48.722310 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.722284    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-data-volume\") pod \"insights-runtime-extractor-nrxcw\" (UID: \"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88\") " pod="openshift-insights/insights-runtime-extractor-nrxcw"
Sep 16 18:45:48.722457 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.722439    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nrxcw\" (UID: \"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88\") " pod="openshift-insights/insights-runtime-extractor-nrxcw"
Sep 16 18:45:48.722506 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.722483    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtpf5\" (UniqueName: \"kubernetes.io/projected/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-kube-api-access-rtpf5\") pod \"insights-runtime-extractor-nrxcw\" (UID: \"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88\") " pod="openshift-insights/insights-runtime-extractor-nrxcw"
Sep 16 18:45:48.722572 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.722533    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-crio-socket\") pod \"insights-runtime-extractor-nrxcw\" (UID: \"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88\") " pod="openshift-insights/insights-runtime-extractor-nrxcw"
Sep 16 18:45:48.722572 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:48.722536    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:45:48.722684 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.722589    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-crio-socket\") pod \"insights-runtime-extractor-nrxcw\" (UID: \"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88\") " pod="openshift-insights/insights-runtime-extractor-nrxcw"
Sep 16 18:45:48.722684 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:48.722615    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-kube-rbac-proxy-cm podName:3900c7ee-b6e7-4edf-84a3-e44cb8c75b88 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:49.222599642 +0000 UTC m=+59.298369251 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nrxcw" (UID: "3900c7ee-b6e7-4edf-84a3-e44cb8c75b88") : configmap "kube-rbac-proxy" not found
Sep 16 18:45:48.722684 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.722614    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-data-volume\") pod \"insights-runtime-extractor-nrxcw\" (UID: \"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88\") " pod="openshift-insights/insights-runtime-extractor-nrxcw"
Sep 16 18:45:48.722832 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.722737    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nrxcw\" (UID: \"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88\") " pod="openshift-insights/insights-runtime-extractor-nrxcw"
Sep 16 18:45:48.722873 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:48.722831    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:48.722873 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:48.722871    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-insights-runtime-extractor-tls podName:3900c7ee-b6e7-4edf-84a3-e44cb8c75b88 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:49.222857902 +0000 UTC m=+59.298627513 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nrxcw" (UID: "3900c7ee-b6e7-4edf-84a3-e44cb8c75b88") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:48.732439 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.732414    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtpf5\" (UniqueName: \"kubernetes.io/projected/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-kube-api-access-rtpf5\") pod \"insights-runtime-extractor-nrxcw\" (UID: \"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88\") " pod="openshift-insights/insights-runtime-extractor-nrxcw"
Sep 16 18:45:48.892090 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.892032    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-rnfbn"]
Sep 16 18:45:48.893955 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.893941    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rnfbn"
Sep 16 18:45:48.896159 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.896139    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 16 18:45:48.896392 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.896377    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-7f26f\""
Sep 16 18:45:48.896459 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.896378    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 16 18:45:48.924405 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.924385    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvhdh\" (UniqueName: \"kubernetes.io/projected/e0a3e579-db97-47c8-bfd2-544ce39f92de-kube-api-access-xvhdh\") pod \"migrator-8b4d697c5-rnfbn\" (UID: \"e0a3e579-db97-47c8-bfd2-544ce39f92de\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rnfbn"
Sep 16 18:45:48.954244 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:48.954220    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-rnfbn"]
Sep 16 18:45:49.024730 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:49.024693    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvhdh\" (UniqueName: \"kubernetes.io/projected/e0a3e579-db97-47c8-bfd2-544ce39f92de-kube-api-access-xvhdh\") pod \"migrator-8b4d697c5-rnfbn\" (UID: \"e0a3e579-db97-47c8-bfd2-544ce39f92de\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rnfbn"
Sep 16 18:45:49.048273 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:49.047474    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvhdh\" (UniqueName: \"kubernetes.io/projected/e0a3e579-db97-47c8-bfd2-544ce39f92de-kube-api-access-xvhdh\") pod \"migrator-8b4d697c5-rnfbn\" (UID: \"e0a3e579-db97-47c8-bfd2-544ce39f92de\") " pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rnfbn"
Sep 16 18:45:49.202409 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:49.202350    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rnfbn"
Sep 16 18:45:49.225573 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:49.225536    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nrxcw\" (UID: \"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88\") " pod="openshift-insights/insights-runtime-extractor-nrxcw"
Sep 16 18:45:49.225685 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:49.225661    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nrxcw\" (UID: \"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88\") " pod="openshift-insights/insights-runtime-extractor-nrxcw"
Sep 16 18:45:49.225829 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:49.225696    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:49.225829 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:49.225802    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-insights-runtime-extractor-tls podName:3900c7ee-b6e7-4edf-84a3-e44cb8c75b88 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:50.225779169 +0000 UTC m=+60.301548803 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nrxcw" (UID: "3900c7ee-b6e7-4edf-84a3-e44cb8c75b88") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:49.225958 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:49.225824    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:45:49.226006 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:49.225958    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-kube-rbac-proxy-cm podName:3900c7ee-b6e7-4edf-84a3-e44cb8c75b88 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:50.225936453 +0000 UTC m=+60.301706083 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nrxcw" (UID: "3900c7ee-b6e7-4edf-84a3-e44cb8c75b88") : configmap "kube-rbac-proxy" not found
Sep 16 18:45:49.472563 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:49.472495    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-8b4d697c5-rnfbn"]
Sep 16 18:45:49.476669 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:45:49.476636    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a3e579_db97_47c8_bfd2_544ce39f92de.slice/crio-1a809570513869caf24236811ab659ddd3615356f652c08b03cfa8e5c165a8c3 WatchSource:0}: Error finding container 1a809570513869caf24236811ab659ddd3615356f652c08b03cfa8e5c165a8c3: Status 404 returned error can't find the container with id 1a809570513869caf24236811ab659ddd3615356f652c08b03cfa8e5c165a8c3
Sep 16 18:45:49.707441 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:49.707415    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rnfbn" event={"ID":"e0a3e579-db97-47c8-bfd2-544ce39f92de","Type":"ContainerStarted","Data":"1a809570513869caf24236811ab659ddd3615356f652c08b03cfa8e5c165a8c3"}
Sep 16 18:45:50.232179 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:50.232147    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nrxcw\" (UID: \"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88\") " pod="openshift-insights/insights-runtime-extractor-nrxcw"
Sep 16 18:45:50.232351 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:50.232236    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nrxcw\" (UID: \"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88\") " pod="openshift-insights/insights-runtime-extractor-nrxcw"
Sep 16 18:45:50.232424 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:50.232401    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:50.232541 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:50.232476    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-insights-runtime-extractor-tls podName:3900c7ee-b6e7-4edf-84a3-e44cb8c75b88 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:52.232456244 +0000 UTC m=+62.308225856 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nrxcw" (UID: "3900c7ee-b6e7-4edf-84a3-e44cb8c75b88") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:50.232541 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:50.232496    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:45:50.232643 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:50.232559    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-kube-rbac-proxy-cm podName:3900c7ee-b6e7-4edf-84a3-e44cb8c75b88 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:52.232541857 +0000 UTC m=+62.308311482 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nrxcw" (UID: "3900c7ee-b6e7-4edf-84a3-e44cb8c75b88") : configmap "kube-rbac-proxy" not found
Sep 16 18:45:50.626363 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:50.626330    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kdln4"
Sep 16 18:45:50.703300 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:50.703279    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tdn77_8f16d955-3a40-4270-8810-9f50eecea21d/dns-node-resolver/0.log"
Sep 16 18:45:51.138890 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:51.138857    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-ln8nz\" (UID: \"b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ln8nz"
Sep 16 18:45:51.139255 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:51.138995    2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:45:51.139255 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:51.139063    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5-cluster-monitoring-operator-tls podName:b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:59.139047172 +0000 UTC m=+69.214816782 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-ln8nz" (UID: "b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:45:51.340054 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:51.340027    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-registry-tls\") pod \"image-registry-558956f4c-ptzxv\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") " pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:51.340233 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:51.340121    2554 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 16 18:45:51.340233 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:51.340132    2554 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-558956f4c-ptzxv: secret "image-registry-tls" not found
Sep 16 18:45:51.340233 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:51.340173    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-registry-tls podName:2688504b-6527-42f6-a904-8ef74b146bc7 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:59.340160865 +0000 UTC m=+69.415930475 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-registry-tls") pod "image-registry-558956f4c-ptzxv" (UID: "2688504b-6527-42f6-a904-8ef74b146bc7") : secret "image-registry-tls" not found
Sep 16 18:45:51.511177 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:51.511155    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-grtxh_e82386e7-000e-4693-aa95-d54f59be5f8b/node-ca/0.log"
Sep 16 18:45:51.715907 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:51.715875    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rnfbn" event={"ID":"e0a3e579-db97-47c8-bfd2-544ce39f92de","Type":"ContainerStarted","Data":"6da1ed245bfd1838aa28a5b8657445ce878b877d4cf72c4525afac3ae5a3d01c"}
Sep 16 18:45:51.716133 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:51.715977    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rnfbn" event={"ID":"e0a3e579-db97-47c8-bfd2-544ce39f92de","Type":"ContainerStarted","Data":"04c615111d9644c8954ef2e0b4ea715289e784be42df236c4cf2e756c6ea0be2"}
Sep 16 18:45:51.737589 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:51.737540    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-8b4d697c5-rnfbn" podStartSLOduration=2.54533091 podStartE2EDuration="3.737523955s" totalImagesPullingTime="1.192193045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:45:51.736975847 +0000 UTC m=+61.812745479" watchObservedRunningTime="2026-09-16 18:45:51.737523955 +0000 UTC m=+61.813293807"
Sep 16 18:45:52.246373 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:52.246344    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nrxcw\" (UID: \"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88\") " pod="openshift-insights/insights-runtime-extractor-nrxcw"
Sep 16 18:45:52.246752 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:52.246482    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:52.246752 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:52.246500    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nrxcw\" (UID: \"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88\") " pod="openshift-insights/insights-runtime-extractor-nrxcw"
Sep 16 18:45:52.246752 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:52.246540    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-insights-runtime-extractor-tls podName:3900c7ee-b6e7-4edf-84a3-e44cb8c75b88 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:56.246524897 +0000 UTC m=+66.322294506 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nrxcw" (UID: "3900c7ee-b6e7-4edf-84a3-e44cb8c75b88") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:52.246752 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:52.246564    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:45:52.246752 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:52.246620    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-kube-rbac-proxy-cm podName:3900c7ee-b6e7-4edf-84a3-e44cb8c75b88 nodeName:}" failed. No retries permitted until 2026-09-16 18:45:56.246608361 +0000 UTC m=+66.322377972 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nrxcw" (UID: "3900c7ee-b6e7-4edf-84a3-e44cb8c75b88") : configmap "kube-rbac-proxy" not found
Sep 16 18:45:52.303751 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:52.303708    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pbdzg_d7440848-ec21-400a-9143-2b6794067e8a/serve-healthcheck-canary/0.log"
Sep 16 18:45:53.503572 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:53.503533    2554 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-llwg5"
Sep 16 18:45:53.503964 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:53.503598    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-llwg5"
Sep 16 18:45:53.503964 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:53.503931    2554 scope.go:117] "RemoveContainer" containerID="c9b969c4b66bedcd5da83befa495bbb975baee6be56ae81e982dd36cedddd6eb"
Sep 16 18:45:53.504093 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:53.504072    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-696884fb9c-llwg5_openshift-console-operator(3589130e-56b7-4ce2-ad13-55804eabd1b3)\"" pod="openshift-console-operator/console-operator-696884fb9c-llwg5" podUID="3589130e-56b7-4ce2-ad13-55804eabd1b3"
Sep 16 18:45:56.175668 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:56.175617    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f2230c8-6009-4227-8dbc-12e10ebe55f2-original-pull-secret\") pod \"global-pull-secret-syncer-4pmm6\" (UID: \"9f2230c8-6009-4227-8dbc-12e10ebe55f2\") " pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:45:56.176121 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:56.175697    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59dba60-7c0d-4e12-b53a-3462d4b9791a-metrics-certs\") pod \"network-metrics-daemon-zp95h\" (UID: \"d59dba60-7c0d-4e12-b53a-3462d4b9791a\") " pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:45:56.178606 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:56.178579    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 16 18:45:56.178754 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:56.178629    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 16 18:45:56.186805 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:56.186786    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 16 18:45:56.186906 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:56.186856    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d59dba60-7c0d-4e12-b53a-3462d4b9791a-metrics-certs podName:d59dba60-7c0d-4e12-b53a-3462d4b9791a nodeName:}" failed. No retries permitted until 2026-09-16 18:47:00.186833954 +0000 UTC m=+130.262603580 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d59dba60-7c0d-4e12-b53a-3462d4b9791a-metrics-certs") pod "network-metrics-daemon-zp95h" (UID: "d59dba60-7c0d-4e12-b53a-3462d4b9791a") : secret "metrics-daemon-secret" not found
Sep 16 18:45:56.188970 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:56.188948    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9f2230c8-6009-4227-8dbc-12e10ebe55f2-original-pull-secret\") pod \"global-pull-secret-syncer-4pmm6\" (UID: \"9f2230c8-6009-4227-8dbc-12e10ebe55f2\") " pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:45:56.276396 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:56.276371    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nrxcw\" (UID: \"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88\") " pod="openshift-insights/insights-runtime-extractor-nrxcw"
Sep 16 18:45:56.276483 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:56.276404    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jn69\" (UniqueName: \"kubernetes.io/projected/b17c052d-1029-47d7-960f-af161a00f71c-kube-api-access-2jn69\") pod \"network-check-target-dvtnm\" (UID: \"b17c052d-1029-47d7-960f-af161a00f71c\") " pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:45:56.276542 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:56.276520    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:56.276577 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:56.276559    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nrxcw\" (UID: \"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88\") " pod="openshift-insights/insights-runtime-extractor-nrxcw"
Sep 16 18:45:56.276612 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:56.276591    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-insights-runtime-extractor-tls podName:3900c7ee-b6e7-4edf-84a3-e44cb8c75b88 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:04.276572339 +0000 UTC m=+74.352341951 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nrxcw" (UID: "3900c7ee-b6e7-4edf-84a3-e44cb8c75b88") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:45:56.276654 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:56.276617    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 16 18:45:56.276687 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:56.276657    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-kube-rbac-proxy-cm podName:3900c7ee-b6e7-4edf-84a3-e44cb8c75b88 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:04.276645087 +0000 UTC m=+74.352414727 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nrxcw" (UID: "3900c7ee-b6e7-4edf-84a3-e44cb8c75b88") : configmap "kube-rbac-proxy" not found
Sep 16 18:45:56.278609 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:56.278590    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 16 18:45:56.289212 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:56.289190    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 16 18:45:56.299710 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:56.299676    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jn69\" (UniqueName: \"kubernetes.io/projected/b17c052d-1029-47d7-960f-af161a00f71c-kube-api-access-2jn69\") pod \"network-check-target-dvtnm\" (UID: \"b17c052d-1029-47d7-960f-af161a00f71c\") " pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:45:56.457137 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:56.457074    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4pmm6"
Sep 16 18:45:56.465414 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:56.465395    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2qmgm\""
Sep 16 18:45:56.473601 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:56.473580    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:45:56.579338 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:56.579307    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-metrics-tls\") pod \"dns-default-tjwj4\" (UID: \"d0bc4f19-10ce-472d-b31b-dd58bb9d7afa\") " pod="openshift-dns/dns-default-tjwj4"
Sep 16 18:45:56.579480 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:56.579460    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 16 18:45:56.579531 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:56.579520    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-metrics-tls podName:d0bc4f19-10ce-472d-b31b-dd58bb9d7afa nodeName:}" failed. No retries permitted until 2026-09-16 18:46:28.579504762 +0000 UTC m=+98.655274391 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-metrics-tls") pod "dns-default-tjwj4" (UID: "d0bc4f19-10ce-472d-b31b-dd58bb9d7afa") : secret "dns-default-metrics-tls" not found
Sep 16 18:45:56.732538 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:56.732476    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4pmm6"]
Sep 16 18:45:56.735952 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:45:56.735930    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f2230c8_6009_4227_8dbc_12e10ebe55f2.slice/crio-09f1e6ecf917c0c17cf532533b3ea944b1f3ec358a8b9774ecb2721234152fe2 WatchSource:0}: Error finding container 09f1e6ecf917c0c17cf532533b3ea944b1f3ec358a8b9774ecb2721234152fe2: Status 404 returned error can't find the container with id 09f1e6ecf917c0c17cf532533b3ea944b1f3ec358a8b9774ecb2721234152fe2
Sep 16 18:45:56.756098 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:56.756069    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dvtnm"]
Sep 16 18:45:56.758494 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:45:56.758472    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb17c052d_1029_47d7_960f_af161a00f71c.slice/crio-ee24a6c1d04d54f0fccf906375716e2ef6e86582cf98ba71680cd0e634d05f2d WatchSource:0}: Error finding container ee24a6c1d04d54f0fccf906375716e2ef6e86582cf98ba71680cd0e634d05f2d: Status 404 returned error can't find the container with id ee24a6c1d04d54f0fccf906375716e2ef6e86582cf98ba71680cd0e634d05f2d
Sep 16 18:45:57.742117 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:57.742075    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dvtnm" event={"ID":"b17c052d-1029-47d7-960f-af161a00f71c","Type":"ContainerStarted","Data":"ee24a6c1d04d54f0fccf906375716e2ef6e86582cf98ba71680cd0e634d05f2d"}
Sep 16 18:45:57.743662 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:57.743628    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4pmm6" event={"ID":"9f2230c8-6009-4227-8dbc-12e10ebe55f2","Type":"ContainerStarted","Data":"09f1e6ecf917c0c17cf532533b3ea944b1f3ec358a8b9774ecb2721234152fe2"}
Sep 16 18:45:59.202025 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:59.201989    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-ln8nz\" (UID: \"b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ln8nz"
Sep 16 18:45:59.202487 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:59.202128    2554 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 16 18:45:59.202487 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:45:59.202196    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5-cluster-monitoring-operator-tls podName:b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:15.202176225 +0000 UTC m=+85.277945854 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cd954b897-ln8nz" (UID: "b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5") : secret "cluster-monitoring-operator-tls" not found
Sep 16 18:45:59.403543 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:59.403509    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-registry-tls\") pod \"image-registry-558956f4c-ptzxv\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") " pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:59.406217 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:59.406194    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-registry-tls\") pod \"image-registry-558956f4c-ptzxv\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") " pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:45:59.557033 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:59.557002    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hvbtm\""
Sep 16 18:45:59.565039 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:45:59.565015    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:46:01.759057 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:01.759019    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4pmm6" event={"ID":"9f2230c8-6009-4227-8dbc-12e10ebe55f2","Type":"ContainerStarted","Data":"d85613f711529d6719033e977f247953171cd1e58df39cd7dc420d91dba24b52"}
Sep 16 18:46:01.764479 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:01.764449    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dvtnm" event={"ID":"b17c052d-1029-47d7-960f-af161a00f71c","Type":"ContainerStarted","Data":"cbd6dfef232041d67a31a087b9545a3d45e45355b4de6d20910ec63f1f3e4db1"}
Sep 16 18:46:01.764656 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:01.764640    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:46:01.776758 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:01.776695    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4pmm6" podStartSLOduration=65.951040036 podStartE2EDuration="1m10.776682527s" totalImagesPullingTime="4.825642491s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:44:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:01.775430744 +0000 UTC m=+71.851200376" watchObservedRunningTime="2026-09-16 18:46:01.776682527 +0000 UTC m=+71.852452159"
Sep 16 18:46:01.797921 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:01.797847    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dvtnm" podStartSLOduration=66.009897466 podStartE2EDuration="1m10.797831717s" totalImagesPullingTime="4.787934251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:01.79624307 +0000 UTC m=+71.872012703" watchObservedRunningTime="2026-09-16 18:46:01.797831717 +0000 UTC m=+71.873601350"
Sep 16 18:46:01.820500 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:01.820470    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-558956f4c-ptzxv"]
Sep 16 18:46:01.822885 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:46:01.822859    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2688504b_6527_42f6_a904_8ef74b146bc7.slice/crio-903bc1b8a23d49794a92046ee9b28282f09d36a351da24a099fb710229f30981 WatchSource:0}: Error finding container 903bc1b8a23d49794a92046ee9b28282f09d36a351da24a099fb710229f30981: Status 404 returned error can't find the container with id 903bc1b8a23d49794a92046ee9b28282f09d36a351da24a099fb710229f30981
Sep 16 18:46:02.769604 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:02.769553    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-558956f4c-ptzxv" event={"ID":"2688504b-6527-42f6-a904-8ef74b146bc7","Type":"ContainerStarted","Data":"084aafd0b98d36d9fd0b97352ddf7bfd8e49ffebbcdfdcb4676e65e62cb53600"}
Sep 16 18:46:02.770076 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:02.769622    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-558956f4c-ptzxv" event={"ID":"2688504b-6527-42f6-a904-8ef74b146bc7","Type":"ContainerStarted","Data":"903bc1b8a23d49794a92046ee9b28282f09d36a351da24a099fb710229f30981"}
Sep 16 18:46:02.770076 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:02.769874    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:46:02.797673 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:02.797615    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-558956f4c-ptzxv" podStartSLOduration=19.797600156 podStartE2EDuration="19.797600156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:02.797117301 +0000 UTC m=+72.872886934" watchObservedRunningTime="2026-09-16 18:46:02.797600156 +0000 UTC m=+72.873369789"
Sep 16 18:46:04.337536 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:04.337499    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nrxcw\" (UID: \"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88\") " pod="openshift-insights/insights-runtime-extractor-nrxcw"
Sep 16 18:46:04.337984 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:04.337558    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nrxcw\" (UID: \"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88\") " pod="openshift-insights/insights-runtime-extractor-nrxcw"
Sep 16 18:46:04.337984 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:46:04.337648    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:04.337984 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:46:04.337698    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-insights-runtime-extractor-tls podName:3900c7ee-b6e7-4edf-84a3-e44cb8c75b88 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:20.337684457 +0000 UTC m=+90.413454067 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nrxcw" (UID: "3900c7ee-b6e7-4edf-84a3-e44cb8c75b88") : secret "insights-runtime-extractor-tls" not found
Sep 16 18:46:04.338097 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:04.338040    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nrxcw\" (UID: \"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88\") " pod="openshift-insights/insights-runtime-extractor-nrxcw"
Sep 16 18:46:07.438704 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:07.438669    2554 scope.go:117] "RemoveContainer" containerID="c9b969c4b66bedcd5da83befa495bbb975baee6be56ae81e982dd36cedddd6eb"
Sep 16 18:46:07.787267 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:07.787240    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/2.log"
Sep 16 18:46:07.787766 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:07.787748    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/1.log"
Sep 16 18:46:07.787873 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:07.787791    2554 generic.go:356] "Generic (PLEG): container finished" podID="3589130e-56b7-4ce2-ad13-55804eabd1b3" containerID="fe5696b3ea2ca860945e6aa76576bd547820abebb73e64bb2088dfc32c41b21e" exitCode=255
Sep 16 18:46:07.787873 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:07.787832    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-llwg5" event={"ID":"3589130e-56b7-4ce2-ad13-55804eabd1b3","Type":"ContainerDied","Data":"fe5696b3ea2ca860945e6aa76576bd547820abebb73e64bb2088dfc32c41b21e"}
Sep 16 18:46:07.787978 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:07.787895    2554 scope.go:117] "RemoveContainer" containerID="c9b969c4b66bedcd5da83befa495bbb975baee6be56ae81e982dd36cedddd6eb"
Sep 16 18:46:07.788188 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:07.788169    2554 scope.go:117] "RemoveContainer" containerID="fe5696b3ea2ca860945e6aa76576bd547820abebb73e64bb2088dfc32c41b21e"
Sep 16 18:46:07.788367 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:46:07.788342    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-llwg5_openshift-console-operator(3589130e-56b7-4ce2-ad13-55804eabd1b3)\"" pod="openshift-console-operator/console-operator-696884fb9c-llwg5" podUID="3589130e-56b7-4ce2-ad13-55804eabd1b3"
Sep 16 18:46:08.792314 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:08.792283    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/2.log"
Sep 16 18:46:13.503215 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:13.503087    2554 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-696884fb9c-llwg5"
Sep 16 18:46:13.503215 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:13.503148    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-llwg5"
Sep 16 18:46:13.503638 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:13.503419    2554 scope.go:117] "RemoveContainer" containerID="fe5696b3ea2ca860945e6aa76576bd547820abebb73e64bb2088dfc32c41b21e"
Sep 16 18:46:13.503638 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:46:13.503569    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-llwg5_openshift-console-operator(3589130e-56b7-4ce2-ad13-55804eabd1b3)\"" pod="openshift-console-operator/console-operator-696884fb9c-llwg5" podUID="3589130e-56b7-4ce2-ad13-55804eabd1b3"
Sep 16 18:46:15.210318 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:15.210272    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-ln8nz\" (UID: \"b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ln8nz"
Sep 16 18:46:15.213046 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:15.213011    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cd954b897-ln8nz\" (UID: \"b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5\") " pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ln8nz"
Sep 16 18:46:15.256065 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:15.256020    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-558956f4c-ptzxv"]
Sep 16 18:46:15.512252 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:15.512228    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-w9rhs\""
Sep 16 18:46:15.521332 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:15.521314    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ln8nz"
Sep 16 18:46:15.792425 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:15.792362    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cd954b897-ln8nz"]
Sep 16 18:46:15.796889 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:46:15.796865    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb62bbe6f_7dcd_4295_a2d8_78d1dd7036b5.slice/crio-fdf6bddc2d191da0f242cb10dbdf8de1ab48915da26063cab4eddfd068b1ec23 WatchSource:0}: Error finding container fdf6bddc2d191da0f242cb10dbdf8de1ab48915da26063cab4eddfd068b1ec23: Status 404 returned error can't find the container with id fdf6bddc2d191da0f242cb10dbdf8de1ab48915da26063cab4eddfd068b1ec23
Sep 16 18:46:15.814321 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:15.814292    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ln8nz" event={"ID":"b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5","Type":"ContainerStarted","Data":"fdf6bddc2d191da0f242cb10dbdf8de1ab48915da26063cab4eddfd068b1ec23"}
Sep 16 18:46:18.688752 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:18.688710    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-zn2th"]
Sep 16 18:46:18.691007 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:18.690990    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-zn2th"
Sep 16 18:46:18.693171 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:18.693152    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 16 18:46:18.694028 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:18.694006    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7bbvc\""
Sep 16 18:46:18.711800 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:18.711781    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-zn2th"]
Sep 16 18:46:18.735358 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:18.735336    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b480ee02-4a14-4111-84f6-21920933edba-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-zn2th\" (UID: \"b480ee02-4a14-4111-84f6-21920933edba\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-zn2th"
Sep 16 18:46:18.824137 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:18.824108    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ln8nz" event={"ID":"b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5","Type":"ContainerStarted","Data":"c2092d002d5d4ce9fca6db700320ff692e56470766b06ee775b4d9b01ee220ac"}
Sep 16 18:46:18.835614 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:18.835590    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b480ee02-4a14-4111-84f6-21920933edba-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-zn2th\" (UID: \"b480ee02-4a14-4111-84f6-21920933edba\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-zn2th"
Sep 16 18:46:18.835737 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:46:18.835709    2554 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 18:46:18.835790 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:46:18.835776    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b480ee02-4a14-4111-84f6-21920933edba-tls-certificates podName:b480ee02-4a14-4111-84f6-21920933edba nodeName:}" failed. No retries permitted until 2026-09-16 18:46:19.335760845 +0000 UTC m=+89.411530455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/b480ee02-4a14-4111-84f6-21920933edba-tls-certificates") pod "prometheus-operator-admission-webhook-5864cd99d7-zn2th" (UID: "b480ee02-4a14-4111-84f6-21920933edba") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 16 18:46:18.846944 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:18.846910    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cd954b897-ln8nz" podStartSLOduration=33.576049565 podStartE2EDuration="35.846893594s" totalImagesPullingTime="2.270844029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:18.845848782 +0000 UTC m=+88.921618413" watchObservedRunningTime="2026-09-16 18:46:18.846893594 +0000 UTC m=+88.922663244"
Sep 16 18:46:19.340267 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:19.340227    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b480ee02-4a14-4111-84f6-21920933edba-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-zn2th\" (UID: \"b480ee02-4a14-4111-84f6-21920933edba\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-zn2th"
Sep 16 18:46:19.342703 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:19.342678    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b480ee02-4a14-4111-84f6-21920933edba-tls-certificates\") pod \"prometheus-operator-admission-webhook-5864cd99d7-zn2th\" (UID: \"b480ee02-4a14-4111-84f6-21920933edba\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-zn2th"
Sep 16 18:46:19.599281 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:19.599181    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-zn2th"
Sep 16 18:46:19.864664 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:19.864560    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-zn2th"]
Sep 16 18:46:19.867005 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:46:19.866975    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb480ee02_4a14_4111_84f6_21920933edba.slice/crio-234355a26e41170f0cef575c14ba042bc28c77bd7bf985396409c889e7b066e2 WatchSource:0}: Error finding container 234355a26e41170f0cef575c14ba042bc28c77bd7bf985396409c889e7b066e2: Status 404 returned error can't find the container with id 234355a26e41170f0cef575c14ba042bc28c77bd7bf985396409c889e7b066e2
Sep 16 18:46:20.347431 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:20.347404    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nrxcw\" (UID: \"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88\") " pod="openshift-insights/insights-runtime-extractor-nrxcw"
Sep 16 18:46:20.349592 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:20.349572    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3900c7ee-b6e7-4edf-84a3-e44cb8c75b88-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nrxcw\" (UID: \"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88\") " pod="openshift-insights/insights-runtime-extractor-nrxcw"
Sep 16 18:46:20.388598 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:20.388579    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-sppxv\""
Sep 16 18:46:20.396845 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:20.396829    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nrxcw"
Sep 16 18:46:20.666406 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:20.666115    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nrxcw"]
Sep 16 18:46:20.668960 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:46:20.668932    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3900c7ee_b6e7_4edf_84a3_e44cb8c75b88.slice/crio-586765a676f09bc6be6352503082291d7f1b42e6217bec5972e0efb7b6131600 WatchSource:0}: Error finding container 586765a676f09bc6be6352503082291d7f1b42e6217bec5972e0efb7b6131600: Status 404 returned error can't find the container with id 586765a676f09bc6be6352503082291d7f1b42e6217bec5972e0efb7b6131600
Sep 16 18:46:20.833774 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:20.833736    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nrxcw" event={"ID":"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88","Type":"ContainerStarted","Data":"dd741764d385331839c1fb671a6be35616e257d335a9eb18e77034b53d95a3c1"}
Sep 16 18:46:20.833939 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:20.833804    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nrxcw" event={"ID":"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88","Type":"ContainerStarted","Data":"586765a676f09bc6be6352503082291d7f1b42e6217bec5972e0efb7b6131600"}
Sep 16 18:46:20.835055 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:20.835019    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-zn2th" event={"ID":"b480ee02-4a14-4111-84f6-21920933edba","Type":"ContainerStarted","Data":"234355a26e41170f0cef575c14ba042bc28c77bd7bf985396409c889e7b066e2"}
Sep 16 18:46:21.839888 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:21.839856    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nrxcw" event={"ID":"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88","Type":"ContainerStarted","Data":"a21972cee76b9a3f0e5a3ba1048a28b1990e3d27834a83a6aef5bef6712bdd17"}
Sep 16 18:46:21.841305 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:21.841282    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-zn2th" event={"ID":"b480ee02-4a14-4111-84f6-21920933edba","Type":"ContainerStarted","Data":"357bd2358acbcd5562279f40a335ca0905042d16d78322e813e5f02a1a2389c7"}
Sep 16 18:46:21.841428 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:21.841406    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-zn2th"
Sep 16 18:46:21.845837 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:21.845820    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-zn2th"
Sep 16 18:46:21.897968 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:21.897928    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5864cd99d7-zn2th" podStartSLOduration=2.762106761 podStartE2EDuration="3.897915683s" totalImagesPullingTime="1.135808922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:21.871615478 +0000 UTC m=+91.947385110" watchObservedRunningTime="2026-09-16 18:46:21.897915683 +0000 UTC m=+91.973685314"
Sep 16 18:46:22.846206 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:22.846180    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nrxcw" event={"ID":"3900c7ee-b6e7-4edf-84a3-e44cb8c75b88","Type":"ContainerStarted","Data":"5708a8c276f699e64220ab275f84a85b0bf1ed2110f184299a65aec798413ed1"}
Sep 16 18:46:22.869211 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:22.869173    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nrxcw" podStartSLOduration=32.863627573 podStartE2EDuration="34.869158742s" totalImagesPullingTime="2.005531169s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 18:45:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:22.868629573 +0000 UTC m=+92.944399204" watchObservedRunningTime="2026-09-16 18:46:22.869158742 +0000 UTC m=+92.944928372"
Sep 16 18:46:25.262271 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:25.262247    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:46:25.438196 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:25.438172    2554 scope.go:117] "RemoveContainer" containerID="fe5696b3ea2ca860945e6aa76576bd547820abebb73e64bb2088dfc32c41b21e"
Sep 16 18:46:25.438327 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:46:25.438299    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-696884fb9c-llwg5_openshift-console-operator(3589130e-56b7-4ce2-ad13-55804eabd1b3)\"" pod="openshift-console-operator/console-operator-696884fb9c-llwg5" podUID="3589130e-56b7-4ce2-ad13-55804eabd1b3"
Sep 16 18:46:27.129088 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.129056    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-s5zl7"]
Sep 16 18:46:27.132902 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.132887    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-s5zl7"
Sep 16 18:46:27.134663 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.134635    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Sep 16 18:46:27.134814 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.134738    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Sep 16 18:46:27.134981 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.134964    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-85xqg\""
Sep 16 18:46:27.135397 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.135377    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 16 18:46:27.135475 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.135424    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Sep 16 18:46:27.139554 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.139533    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9z9fb"]
Sep 16 18:46:27.142910 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.142892    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9z9fb"
Sep 16 18:46:27.148290 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.148202    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-s5zl7"]
Sep 16 18:46:27.150080 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.150061    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 16 18:46:27.150593 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.150569    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 16 18:46:27.153732 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.150810    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 16 18:46:27.153732 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.150822    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pzlzb\""
Sep 16 18:46:27.192841 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.192821    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c745eb77-57e1-4567-b68f-01e301940170-sys\") pod \"node-exporter-9z9fb\" (UID: \"c745eb77-57e1-4567-b68f-01e301940170\") " pod="openshift-monitoring/node-exporter-9z9fb"
Sep 16 18:46:27.192944 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.192856    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c745eb77-57e1-4567-b68f-01e301940170-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9z9fb\" (UID: \"c745eb77-57e1-4567-b68f-01e301940170\") " pod="openshift-monitoring/node-exporter-9z9fb"
Sep 16 18:46:27.192944 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.192874    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6239a312-62d8-4ea0-b52a-d96f7a4d233c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-s5zl7\" (UID: \"6239a312-62d8-4ea0-b52a-d96f7a4d233c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-s5zl7"
Sep 16 18:46:27.192944 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.192889    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c745eb77-57e1-4567-b68f-01e301940170-node-exporter-tls\") pod \"node-exporter-9z9fb\" (UID: \"c745eb77-57e1-4567-b68f-01e301940170\") " pod="openshift-monitoring/node-exporter-9z9fb"
Sep 16 18:46:27.192944 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.192917    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6239a312-62d8-4ea0-b52a-d96f7a4d233c-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-s5zl7\" (UID: \"6239a312-62d8-4ea0-b52a-d96f7a4d233c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-s5zl7"
Sep 16 18:46:27.193119 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.192978    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c745eb77-57e1-4567-b68f-01e301940170-metrics-client-ca\") pod \"node-exporter-9z9fb\" (UID: \"c745eb77-57e1-4567-b68f-01e301940170\") " pod="openshift-monitoring/node-exporter-9z9fb"
Sep 16 18:46:27.193119 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.193050    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6239a312-62d8-4ea0-b52a-d96f7a4d233c-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-s5zl7\" (UID: \"6239a312-62d8-4ea0-b52a-d96f7a4d233c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-s5zl7"
Sep 16 18:46:27.193119 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.193091    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzkhx\" (UniqueName: \"kubernetes.io/projected/c745eb77-57e1-4567-b68f-01e301940170-kube-api-access-xzkhx\") pod \"node-exporter-9z9fb\" (UID: \"c745eb77-57e1-4567-b68f-01e301940170\") " pod="openshift-monitoring/node-exporter-9z9fb"
Sep 16 18:46:27.193119 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.193113    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c745eb77-57e1-4567-b68f-01e301940170-node-exporter-wtmp\") pod \"node-exporter-9z9fb\" (UID: \"c745eb77-57e1-4567-b68f-01e301940170\") " pod="openshift-monitoring/node-exporter-9z9fb"
Sep 16 18:46:27.193269 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.193137    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6239a312-62d8-4ea0-b52a-d96f7a4d233c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-s5zl7\" (UID: \"6239a312-62d8-4ea0-b52a-d96f7a4d233c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-s5zl7"
Sep 16 18:46:27.193269 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.193152    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpjc9\" (UniqueName: \"kubernetes.io/projected/6239a312-62d8-4ea0-b52a-d96f7a4d233c-kube-api-access-xpjc9\") pod \"kube-state-metrics-59fbdd77d-s5zl7\" (UID: \"6239a312-62d8-4ea0-b52a-d96f7a4d233c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-s5zl7"
Sep 16 18:46:27.193269 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.193189    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c745eb77-57e1-4567-b68f-01e301940170-root\") pod \"node-exporter-9z9fb\" (UID: \"c745eb77-57e1-4567-b68f-01e301940170\") " pod="openshift-monitoring/node-exporter-9z9fb"
Sep 16 18:46:27.193269 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.193209    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c745eb77-57e1-4567-b68f-01e301940170-node-exporter-accelerators-collector-config\") pod \"node-exporter-9z9fb\" (UID: \"c745eb77-57e1-4567-b68f-01e301940170\") " pod="openshift-monitoring/node-exporter-9z9fb"
Sep 16 18:46:27.193269 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.193226    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c745eb77-57e1-4567-b68f-01e301940170-node-exporter-textfile\") pod \"node-exporter-9z9fb\" (UID: \"c745eb77-57e1-4567-b68f-01e301940170\") " pod="openshift-monitoring/node-exporter-9z9fb"
Sep 16 18:46:27.193269 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.193243    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6239a312-62d8-4ea0-b52a-d96f7a4d233c-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-s5zl7\" (UID: \"6239a312-62d8-4ea0-b52a-d96f7a4d233c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-s5zl7"
Sep 16 18:46:27.294004 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.293986    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c745eb77-57e1-4567-b68f-01e301940170-node-exporter-wtmp\") pod \"node-exporter-9z9fb\" (UID: \"c745eb77-57e1-4567-b68f-01e301940170\") " pod="openshift-monitoring/node-exporter-9z9fb"
Sep 16 18:46:27.294104 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.294016    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6239a312-62d8-4ea0-b52a-d96f7a4d233c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-s5zl7\" (UID: \"6239a312-62d8-4ea0-b52a-d96f7a4d233c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-s5zl7"
Sep 16 18:46:27.294104 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.294035    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpjc9\" (UniqueName: \"kubernetes.io/projected/6239a312-62d8-4ea0-b52a-d96f7a4d233c-kube-api-access-xpjc9\") pod \"kube-state-metrics-59fbdd77d-s5zl7\" (UID: \"6239a312-62d8-4ea0-b52a-d96f7a4d233c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-s5zl7"
Sep 16 18:46:27.294104 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.294063    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c745eb77-57e1-4567-b68f-01e301940170-root\") pod \"node-exporter-9z9fb\" (UID: \"c745eb77-57e1-4567-b68f-01e301940170\") " pod="openshift-monitoring/node-exporter-9z9fb"
Sep 16 18:46:27.294262 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.294114    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c745eb77-57e1-4567-b68f-01e301940170-node-exporter-accelerators-collector-config\") pod \"node-exporter-9z9fb\" (UID: \"c745eb77-57e1-4567-b68f-01e301940170\") " pod="openshift-monitoring/node-exporter-9z9fb"
Sep 16 18:46:27.294262 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.294144    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c745eb77-57e1-4567-b68f-01e301940170-node-exporter-textfile\") pod \"node-exporter-9z9fb\" (UID: \"c745eb77-57e1-4567-b68f-01e301940170\") " pod="openshift-monitoring/node-exporter-9z9fb"
Sep 16 18:46:27.294262 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.294150    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c745eb77-57e1-4567-b68f-01e301940170-node-exporter-wtmp\") pod \"node-exporter-9z9fb\" (UID: \"c745eb77-57e1-4567-b68f-01e301940170\") " pod="openshift-monitoring/node-exporter-9z9fb"
Sep 16 18:46:27.294262 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.294169    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6239a312-62d8-4ea0-b52a-d96f7a4d233c-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-s5zl7\" (UID: \"6239a312-62d8-4ea0-b52a-d96f7a4d233c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-s5zl7"
Sep 16 18:46:27.294262 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.294219    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c745eb77-57e1-4567-b68f-01e301940170-sys\") pod \"node-exporter-9z9fb\" (UID: \"c745eb77-57e1-4567-b68f-01e301940170\") " pod="openshift-monitoring/node-exporter-9z9fb"
Sep 16 18:46:27.294501 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.294281    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c745eb77-57e1-4567-b68f-01e301940170-sys\") pod \"node-exporter-9z9fb\" (UID: \"c745eb77-57e1-4567-b68f-01e301940170\") " pod="openshift-monitoring/node-exporter-9z9fb"
Sep 16 18:46:27.294501 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:46:27.294299    2554 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Sep 16 18:46:27.294501 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.294323    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c745eb77-57e1-4567-b68f-01e301940170-root\") pod \"node-exporter-9z9fb\" (UID: \"c745eb77-57e1-4567-b68f-01e301940170\") " pod="openshift-monitoring/node-exporter-9z9fb"
Sep 16 18:46:27.294501 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:46:27.294367    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6239a312-62d8-4ea0-b52a-d96f7a4d233c-kube-state-metrics-tls podName:6239a312-62d8-4ea0-b52a-d96f7a4d233c nodeName:}" failed. No retries permitted until 2026-09-16 18:46:27.794349174 +0000 UTC m=+97.870118805 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/6239a312-62d8-4ea0-b52a-d96f7a4d233c-kube-state-metrics-tls") pod "kube-state-metrics-59fbdd77d-s5zl7" (UID: "6239a312-62d8-4ea0-b52a-d96f7a4d233c") : secret "kube-state-metrics-tls" not found
Sep 16 18:46:27.294501 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.294389    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c745eb77-57e1-4567-b68f-01e301940170-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9z9fb\" (UID: \"c745eb77-57e1-4567-b68f-01e301940170\") " pod="openshift-monitoring/node-exporter-9z9fb"
Sep 16 18:46:27.294501 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.294419    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6239a312-62d8-4ea0-b52a-d96f7a4d233c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-s5zl7\" (UID: \"6239a312-62d8-4ea0-b52a-d96f7a4d233c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-s5zl7"
Sep 16 18:46:27.294501 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.294465    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c745eb77-57e1-4567-b68f-01e301940170-node-exporter-textfile\") pod \"node-exporter-9z9fb\" (UID: \"c745eb77-57e1-4567-b68f-01e301940170\") " pod="openshift-monitoring/node-exporter-9z9fb"
Sep 16 18:46:27.294873 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.294515    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c745eb77-57e1-4567-b68f-01e301940170-node-exporter-tls\") pod \"node-exporter-9z9fb\" (UID: \"c745eb77-57e1-4567-b68f-01e301940170\") " pod="openshift-monitoring/node-exporter-9z9fb"
Sep 16 18:46:27.294873 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.294558    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6239a312-62d8-4ea0-b52a-d96f7a4d233c-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-s5zl7\" (UID: \"6239a312-62d8-4ea0-b52a-d96f7a4d233c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-s5zl7"
Sep 16 18:46:27.294873 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.294582    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c745eb77-57e1-4567-b68f-01e301940170-metrics-client-ca\") pod \"node-exporter-9z9fb\" (UID: \"c745eb77-57e1-4567-b68f-01e301940170\") " pod="openshift-monitoring/node-exporter-9z9fb"
Sep 16 18:46:27.294873 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.294609    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6239a312-62d8-4ea0-b52a-d96f7a4d233c-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-s5zl7\" (UID: \"6239a312-62d8-4ea0-b52a-d96f7a4d233c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-s5zl7"
Sep 16 18:46:27.294873 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.294637    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzkhx\" (UniqueName: \"kubernetes.io/projected/c745eb77-57e1-4567-b68f-01e301940170-kube-api-access-xzkhx\") pod \"node-exporter-9z9fb\" (UID: \"c745eb77-57e1-4567-b68f-01e301940170\") " pod="openshift-monitoring/node-exporter-9z9fb"
Sep 16 18:46:27.294873 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.294768    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c745eb77-57e1-4567-b68f-01e301940170-node-exporter-accelerators-collector-config\") pod \"node-exporter-9z9fb\" (UID: \"c745eb77-57e1-4567-b68f-01e301940170\") " pod="openshift-monitoring/node-exporter-9z9fb"
Sep 16 18:46:27.294873 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.294797    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/6239a312-62d8-4ea0-b52a-d96f7a4d233c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-s5zl7\" (UID: \"6239a312-62d8-4ea0-b52a-d96f7a4d233c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-s5zl7"
Sep 16 18:46:27.295264 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.294892    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/6239a312-62d8-4ea0-b52a-d96f7a4d233c-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-s5zl7\" (UID: \"6239a312-62d8-4ea0-b52a-d96f7a4d233c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-s5zl7"
Sep 16 18:46:27.295264 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.295214    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c745eb77-57e1-4567-b68f-01e301940170-metrics-client-ca\") pod \"node-exporter-9z9fb\" (UID: \"c745eb77-57e1-4567-b68f-01e301940170\") " pod="openshift-monitoring/node-exporter-9z9fb"
Sep 16 18:46:27.295421 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.295401    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6239a312-62d8-4ea0-b52a-d96f7a4d233c-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-s5zl7\" (UID: \"6239a312-62d8-4ea0-b52a-d96f7a4d233c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-s5zl7"
Sep 16 18:46:27.296729 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.296705    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c745eb77-57e1-4567-b68f-01e301940170-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9z9fb\" (UID: \"c745eb77-57e1-4567-b68f-01e301940170\") " pod="openshift-monitoring/node-exporter-9z9fb"
Sep 16 18:46:27.297171 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.297152    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c745eb77-57e1-4567-b68f-01e301940170-node-exporter-tls\") pod \"node-exporter-9z9fb\" (UID: \"c745eb77-57e1-4567-b68f-01e301940170\") " pod="openshift-monitoring/node-exporter-9z9fb"
Sep 16 18:46:27.297232 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.297177    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6239a312-62d8-4ea0-b52a-d96f7a4d233c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-s5zl7\" (UID: \"6239a312-62d8-4ea0-b52a-d96f7a4d233c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-s5zl7"
Sep 16 18:46:27.305059 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.305028    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzkhx\" (UniqueName: \"kubernetes.io/projected/c745eb77-57e1-4567-b68f-01e301940170-kube-api-access-xzkhx\") pod \"node-exporter-9z9fb\" (UID: \"c745eb77-57e1-4567-b68f-01e301940170\") " pod="openshift-monitoring/node-exporter-9z9fb"
Sep 16 18:46:27.305166 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.305151    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpjc9\" (UniqueName: \"kubernetes.io/projected/6239a312-62d8-4ea0-b52a-d96f7a4d233c-kube-api-access-xpjc9\") pod \"kube-state-metrics-59fbdd77d-s5zl7\" (UID: \"6239a312-62d8-4ea0-b52a-d96f7a4d233c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-s5zl7"
Sep 16 18:46:27.450737 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.450674    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9z9fb"
Sep 16 18:46:27.462583 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:46:27.462559    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc745eb77_57e1_4567_b68f_01e301940170.slice/crio-b2061d51be73f6db78344fe06b4f248a7cccc08b523701e3b9ebcd7a9aefad89 WatchSource:0}: Error finding container b2061d51be73f6db78344fe06b4f248a7cccc08b523701e3b9ebcd7a9aefad89: Status 404 returned error can't find the container with id b2061d51be73f6db78344fe06b4f248a7cccc08b523701e3b9ebcd7a9aefad89
Sep 16 18:46:27.798846 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.798813    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6239a312-62d8-4ea0-b52a-d96f7a4d233c-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-s5zl7\" (UID: \"6239a312-62d8-4ea0-b52a-d96f7a4d233c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-s5zl7"
Sep 16 18:46:27.801035 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.801018    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6239a312-62d8-4ea0-b52a-d96f7a4d233c-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-s5zl7\" (UID: \"6239a312-62d8-4ea0-b52a-d96f7a4d233c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-s5zl7"
Sep 16 18:46:27.863184 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:27.863150    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9z9fb" event={"ID":"c745eb77-57e1-4567-b68f-01e301940170","Type":"ContainerStarted","Data":"b2061d51be73f6db78344fe06b4f248a7cccc08b523701e3b9ebcd7a9aefad89"}
Sep 16 18:46:28.042022 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.041997    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-s5zl7"
Sep 16 18:46:28.310800 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.310736    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:46:28.316059 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.316039    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.321929 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.321909    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:46:28.322177 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.322162    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:46:28.322682 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.322656    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:46:28.323197 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.323172    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:46:28.323367 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.323338    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jkm5h\""
Sep 16 18:46:28.324267 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.323619    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:46:28.324267 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.323976    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:46:28.324419 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.324334    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:46:28.328016 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.327997    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:46:28.328616 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.328598    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:46:28.349460 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.349437    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:46:28.351371 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.351351    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-s5zl7"]
Sep 16 18:46:28.403884 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.403852    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.404009 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.403896    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.404009 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.403943    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.404009 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.403980    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.404009 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.404004    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.404229 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.404034    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.404229 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.404076    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-web-config\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.404229 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.404102    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.404229 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.404158    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.404229 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.404193    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.404480 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.404231    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-config-out\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.404480 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.404274    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-config-volume\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.404480 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.404305    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrrrz\" (UniqueName: \"kubernetes.io/projected/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-kube-api-access-vrrrz\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.436134 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:46:28.436102    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6239a312_62d8_4ea0_b52a_d96f7a4d233c.slice/crio-0a9764a6ba699ac993eb2be487964469f6f18d48282e3cd8619abadfef1c625e WatchSource:0}: Error finding container 0a9764a6ba699ac993eb2be487964469f6f18d48282e3cd8619abadfef1c625e: Status 404 returned error can't find the container with id 0a9764a6ba699ac993eb2be487964469f6f18d48282e3cd8619abadfef1c625e
Sep 16 18:46:28.504759 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.504737    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrrrz\" (UniqueName: \"kubernetes.io/projected/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-kube-api-access-vrrrz\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.504854 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.504807    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.504854 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.504841    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.504949 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.504881    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.504949 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.504917    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.504949 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:46:28.504920    2554 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 16 18:46:28.504949 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.504940    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.505144 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.504965    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.505144 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.504985    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-web-config\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.505144 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:46:28.505019    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-secret-alertmanager-main-tls podName:3adb4633-501b-4a6d-9a15-9ecbe0068e9f nodeName:}" failed. No retries permitted until 2026-09-16 18:46:29.004997487 +0000 UTC m=+99.080767120 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "3adb4633-501b-4a6d-9a15-9ecbe0068e9f") : secret "alertmanager-main-tls" not found
Sep 16 18:46:28.505144 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.505042    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.505144 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.505077    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.505144 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.505106    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.505144 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.505143    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-config-out\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.505486 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.505183    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-config-volume\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.505486 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.505224    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.506494 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.506465    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.507444 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.507415    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.508100 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.507995    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-web-config\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.508813 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.508783    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.508930 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.508791    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-config-out\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.509114 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.509094    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.509437 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.509414    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.509713 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.509695    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-config-volume\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.510103 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.510086    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.510324 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.510304    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.523052 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.523032    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrrrz\" (UniqueName: \"kubernetes.io/projected/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-kube-api-access-vrrrz\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:28.605758 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.605677    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-metrics-tls\") pod \"dns-default-tjwj4\" (UID: \"d0bc4f19-10ce-472d-b31b-dd58bb9d7afa\") " pod="openshift-dns/dns-default-tjwj4"
Sep 16 18:46:28.607881 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.607857    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d0bc4f19-10ce-472d-b31b-dd58bb9d7afa-metrics-tls\") pod \"dns-default-tjwj4\" (UID: \"d0bc4f19-10ce-472d-b31b-dd58bb9d7afa\") " pod="openshift-dns/dns-default-tjwj4"
Sep 16 18:46:28.612827 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.612803    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6vj57\""
Sep 16 18:46:28.620552 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.620529    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tjwj4"
Sep 16 18:46:28.868103 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.868036    2554 generic.go:356] "Generic (PLEG): container finished" podID="c745eb77-57e1-4567-b68f-01e301940170" containerID="9369a7b38b55e63d3d503c7e1bf2fe82dae702c4d20cac90fca528fbfde61325" exitCode=0
Sep 16 18:46:28.868239 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.868122    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9z9fb" event={"ID":"c745eb77-57e1-4567-b68f-01e301940170","Type":"ContainerDied","Data":"9369a7b38b55e63d3d503c7e1bf2fe82dae702c4d20cac90fca528fbfde61325"}
Sep 16 18:46:28.869413 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.869390    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-s5zl7" event={"ID":"6239a312-62d8-4ea0-b52a-d96f7a4d233c","Type":"ContainerStarted","Data":"0a9764a6ba699ac993eb2be487964469f6f18d48282e3cd8619abadfef1c625e"}
Sep 16 18:46:28.895605 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:28.895581    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tjwj4"]
Sep 16 18:46:28.903784 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:46:28.903754    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0bc4f19_10ce_472d_b31b_dd58bb9d7afa.slice/crio-e1946d2bdb3dc00e152cbc285bb0850d531ef41d18ffb6174530301d9dd4e788 WatchSource:0}: Error finding container e1946d2bdb3dc00e152cbc285bb0850d531ef41d18ffb6174530301d9dd4e788: Status 404 returned error can't find the container with id e1946d2bdb3dc00e152cbc285bb0850d531ef41d18ffb6174530301d9dd4e788
Sep 16 18:46:29.009740 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.009686    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:29.012489 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.012469    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:29.227663 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.227596    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:46:29.383807 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.383755    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-df7545c7c-qwct5"]
Sep 16 18:46:29.388290 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.388240    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5"
Sep 16 18:46:29.395120 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.395100    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 16 18:46:29.395261 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.395218    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 16 18:46:29.395691 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.395671    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 16 18:46:29.396785 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.396760    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-lpzrn\""
Sep 16 18:46:29.397006 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.396988    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 16 18:46:29.397213 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.397144    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 16 18:46:29.401557 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.401309    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2g0eesh09uk52\""
Sep 16 18:46:29.413990 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.413971    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-df7545c7c-qwct5"]
Sep 16 18:46:29.515209 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.515170    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be8ef430-695c-4699-8ed3-46cfc365643e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-df7545c7c-qwct5\" (UID: \"be8ef430-695c-4699-8ed3-46cfc365643e\") " pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5"
Sep 16 18:46:29.515422 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.515232    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/be8ef430-695c-4699-8ed3-46cfc365643e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-df7545c7c-qwct5\" (UID: \"be8ef430-695c-4699-8ed3-46cfc365643e\") " pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5"
Sep 16 18:46:29.515422 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.515254    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/be8ef430-695c-4699-8ed3-46cfc365643e-secret-grpc-tls\") pod \"thanos-querier-df7545c7c-qwct5\" (UID: \"be8ef430-695c-4699-8ed3-46cfc365643e\") " pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5"
Sep 16 18:46:29.515422 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.515328    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be8ef430-695c-4699-8ed3-46cfc365643e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-df7545c7c-qwct5\" (UID: \"be8ef430-695c-4699-8ed3-46cfc365643e\") " pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5"
Sep 16 18:46:29.515422 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.515354    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be8ef430-695c-4699-8ed3-46cfc365643e-metrics-client-ca\") pod \"thanos-querier-df7545c7c-qwct5\" (UID: \"be8ef430-695c-4699-8ed3-46cfc365643e\") " pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5"
Sep 16 18:46:29.515422 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.515389    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/be8ef430-695c-4699-8ed3-46cfc365643e-secret-thanos-querier-tls\") pod \"thanos-querier-df7545c7c-qwct5\" (UID: \"be8ef430-695c-4699-8ed3-46cfc365643e\") " pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5"
Sep 16 18:46:29.515422 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.515418    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vltkr\" (UniqueName: \"kubernetes.io/projected/be8ef430-695c-4699-8ed3-46cfc365643e-kube-api-access-vltkr\") pod \"thanos-querier-df7545c7c-qwct5\" (UID: \"be8ef430-695c-4699-8ed3-46cfc365643e\") " pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5"
Sep 16 18:46:29.515771 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.515444    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/be8ef430-695c-4699-8ed3-46cfc365643e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-df7545c7c-qwct5\" (UID: \"be8ef430-695c-4699-8ed3-46cfc365643e\") " pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5"
Sep 16 18:46:29.518111 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.517087    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:46:29.616177 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.616145    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/be8ef430-695c-4699-8ed3-46cfc365643e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-df7545c7c-qwct5\" (UID: \"be8ef430-695c-4699-8ed3-46cfc365643e\") " pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5"
Sep 16 18:46:29.616335 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.616182    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/be8ef430-695c-4699-8ed3-46cfc365643e-secret-grpc-tls\") pod \"thanos-querier-df7545c7c-qwct5\" (UID: \"be8ef430-695c-4699-8ed3-46cfc365643e\") " pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5"
Sep 16 18:46:29.616335 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.616244    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be8ef430-695c-4699-8ed3-46cfc365643e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-df7545c7c-qwct5\" (UID: \"be8ef430-695c-4699-8ed3-46cfc365643e\") " pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5"
Sep 16 18:46:29.616335 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.616277    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be8ef430-695c-4699-8ed3-46cfc365643e-metrics-client-ca\") pod \"thanos-querier-df7545c7c-qwct5\" (UID: \"be8ef430-695c-4699-8ed3-46cfc365643e\") " pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5"
Sep 16 18:46:29.616335 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.616332    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/be8ef430-695c-4699-8ed3-46cfc365643e-secret-thanos-querier-tls\") pod \"thanos-querier-df7545c7c-qwct5\" (UID: \"be8ef430-695c-4699-8ed3-46cfc365643e\") " pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5"
Sep 16 18:46:29.616646 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.616370    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vltkr\" (UniqueName: \"kubernetes.io/projected/be8ef430-695c-4699-8ed3-46cfc365643e-kube-api-access-vltkr\") pod \"thanos-querier-df7545c7c-qwct5\" (UID: \"be8ef430-695c-4699-8ed3-46cfc365643e\") " pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5"
Sep 16 18:46:29.616646 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.616397    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/be8ef430-695c-4699-8ed3-46cfc365643e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-df7545c7c-qwct5\" (UID: \"be8ef430-695c-4699-8ed3-46cfc365643e\") " pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5"
Sep 16 18:46:29.616646 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.616436    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be8ef430-695c-4699-8ed3-46cfc365643e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-df7545c7c-qwct5\" (UID: \"be8ef430-695c-4699-8ed3-46cfc365643e\") " pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5"
Sep 16 18:46:29.617437 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.617414    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be8ef430-695c-4699-8ed3-46cfc365643e-metrics-client-ca\") pod \"thanos-querier-df7545c7c-qwct5\" (UID: \"be8ef430-695c-4699-8ed3-46cfc365643e\") " pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5"
Sep 16 18:46:29.619855 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.619809    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be8ef430-695c-4699-8ed3-46cfc365643e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-df7545c7c-qwct5\" (UID: \"be8ef430-695c-4699-8ed3-46cfc365643e\") " pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5"
Sep 16 18:46:29.619957 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.619886    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/be8ef430-695c-4699-8ed3-46cfc365643e-secret-grpc-tls\") pod \"thanos-querier-df7545c7c-qwct5\" (UID: \"be8ef430-695c-4699-8ed3-46cfc365643e\") " pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5"
Sep 16 18:46:29.620419 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.620377    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/be8ef430-695c-4699-8ed3-46cfc365643e-secret-thanos-querier-tls\") pod \"thanos-querier-df7545c7c-qwct5\" (UID: \"be8ef430-695c-4699-8ed3-46cfc365643e\") " pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5"
Sep 16 18:46:29.621054 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.621029    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/be8ef430-695c-4699-8ed3-46cfc365643e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-df7545c7c-qwct5\" (UID: \"be8ef430-695c-4699-8ed3-46cfc365643e\") " pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5"
Sep 16 18:46:29.621430 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.621392    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be8ef430-695c-4699-8ed3-46cfc365643e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-df7545c7c-qwct5\" (UID: \"be8ef430-695c-4699-8ed3-46cfc365643e\") " pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5"
Sep 16 18:46:29.621936 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.621912    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/be8ef430-695c-4699-8ed3-46cfc365643e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-df7545c7c-qwct5\" (UID: \"be8ef430-695c-4699-8ed3-46cfc365643e\") " pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5"
Sep 16 18:46:29.627960 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.627920    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vltkr\" (UniqueName: \"kubernetes.io/projected/be8ef430-695c-4699-8ed3-46cfc365643e-kube-api-access-vltkr\") pod \"thanos-querier-df7545c7c-qwct5\" (UID: \"be8ef430-695c-4699-8ed3-46cfc365643e\") " pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5"
Sep 16 18:46:29.699882 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.699852    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5"
Sep 16 18:46:29.796248 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:46:29.796191    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3adb4633_501b_4a6d_9a15_9ecbe0068e9f.slice/crio-c707c2f3403c35054002826ccfbd92ed9d2042c06d11254e65d8553c58a63a7c WatchSource:0}: Error finding container c707c2f3403c35054002826ccfbd92ed9d2042c06d11254e65d8553c58a63a7c: Status 404 returned error can't find the container with id c707c2f3403c35054002826ccfbd92ed9d2042c06d11254e65d8553c58a63a7c
Sep 16 18:46:29.876361 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.876310    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tjwj4" event={"ID":"d0bc4f19-10ce-472d-b31b-dd58bb9d7afa","Type":"ContainerStarted","Data":"e1946d2bdb3dc00e152cbc285bb0850d531ef41d18ffb6174530301d9dd4e788"}
Sep 16 18:46:29.879178 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.879155    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9z9fb" event={"ID":"c745eb77-57e1-4567-b68f-01e301940170","Type":"ContainerStarted","Data":"9cc162ee4232984facf7b57a46a736c4f76f286b616f22c2bd9526d26f2d8d51"}
Sep 16 18:46:29.879286 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.879204    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9z9fb" event={"ID":"c745eb77-57e1-4567-b68f-01e301940170","Type":"ContainerStarted","Data":"d4500d65731c0f59fa39545b0cecfa7d00a18e4b166b9da1bc8c066f7ea191dc"}
Sep 16 18:46:29.880632 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.880605    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3adb4633-501b-4a6d-9a15-9ecbe0068e9f","Type":"ContainerStarted","Data":"c707c2f3403c35054002826ccfbd92ed9d2042c06d11254e65d8553c58a63a7c"}
Sep 16 18:46:29.906060 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:29.906020    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9z9fb" podStartSLOduration=1.825245266 podStartE2EDuration="2.906005012s" totalImagesPullingTime="1.016814332s" totalInitContainerRuntime="63.945414ms" isStatefulPod=true podCreationTimestamp="2026-09-16 18:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:29.903845365 +0000 UTC m=+99.979614996" watchObservedRunningTime="2026-09-16 18:46:29.906005012 +0000 UTC m=+99.981774663"
Sep 16 18:46:30.097115 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:30.097066    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-df7545c7c-qwct5"]
Sep 16 18:46:30.651973 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:46:30.651942    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe8ef430_695c_4699_8ed3_46cfc365643e.slice/crio-9e27edb3838781bdcca853aa2030bb56ef3f515f9e9d1e8ef160929376e8e925 WatchSource:0}: Error finding container 9e27edb3838781bdcca853aa2030bb56ef3f515f9e9d1e8ef160929376e8e925: Status 404 returned error can't find the container with id 9e27edb3838781bdcca853aa2030bb56ef3f515f9e9d1e8ef160929376e8e925
Sep 16 18:46:30.887710 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:30.887675    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-s5zl7" event={"ID":"6239a312-62d8-4ea0-b52a-d96f7a4d233c","Type":"ContainerStarted","Data":"3ffc8ca7e234718f64e9f7d6ff534b3d3867ea46311648433b27da6fe518f8db"}
Sep 16 18:46:30.889016 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:30.888990    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5" event={"ID":"be8ef430-695c-4699-8ed3-46cfc365643e","Type":"ContainerStarted","Data":"9e27edb3838781bdcca853aa2030bb56ef3f515f9e9d1e8ef160929376e8e925"}
Sep 16 18:46:31.562751 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.562709    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-597df9c44c-sqk79"]
Sep 16 18:46:31.583637 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.583606    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-597df9c44c-sqk79"]
Sep 16 18:46:31.583826 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.583758    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-597df9c44c-sqk79"
Sep 16 18:46:31.586537 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.586515    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-cv7eo7r1dhtm1\""
Sep 16 18:46:31.586953 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.586934    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-h9kms\""
Sep 16 18:46:31.587236 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.587216    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 16 18:46:31.587484 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.587467    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 16 18:46:31.588547 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.587693    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 16 18:46:31.588547 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.587915    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 16 18:46:31.736260 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.736193    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/72b0bbb3-86be-47da-a955-334ab24d276c-metrics-server-audit-profiles\") pod \"metrics-server-597df9c44c-sqk79\" (UID: \"72b0bbb3-86be-47da-a955-334ab24d276c\") " pod="openshift-monitoring/metrics-server-597df9c44c-sqk79"
Sep 16 18:46:31.736260 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.736235    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6hd4\" (UniqueName: \"kubernetes.io/projected/72b0bbb3-86be-47da-a955-334ab24d276c-kube-api-access-j6hd4\") pod \"metrics-server-597df9c44c-sqk79\" (UID: \"72b0bbb3-86be-47da-a955-334ab24d276c\") " pod="openshift-monitoring/metrics-server-597df9c44c-sqk79"
Sep 16 18:46:31.736689 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.736271    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/72b0bbb3-86be-47da-a955-334ab24d276c-audit-log\") pod \"metrics-server-597df9c44c-sqk79\" (UID: \"72b0bbb3-86be-47da-a955-334ab24d276c\") " pod="openshift-monitoring/metrics-server-597df9c44c-sqk79"
Sep 16 18:46:31.736689 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.736398    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72b0bbb3-86be-47da-a955-334ab24d276c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-597df9c44c-sqk79\" (UID: \"72b0bbb3-86be-47da-a955-334ab24d276c\") " pod="openshift-monitoring/metrics-server-597df9c44c-sqk79"
Sep 16 18:46:31.736689 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.736494    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/72b0bbb3-86be-47da-a955-334ab24d276c-client-ca-bundle\") pod \"metrics-server-597df9c44c-sqk79\" (UID: \"72b0bbb3-86be-47da-a955-334ab24d276c\") " pod="openshift-monitoring/metrics-server-597df9c44c-sqk79"
Sep 16 18:46:31.736689 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.736528    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/72b0bbb3-86be-47da-a955-334ab24d276c-secret-metrics-server-tls\") pod \"metrics-server-597df9c44c-sqk79\" (UID: \"72b0bbb3-86be-47da-a955-334ab24d276c\") " pod="openshift-monitoring/metrics-server-597df9c44c-sqk79"
Sep 16 18:46:31.736689 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.736560    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/72b0bbb3-86be-47da-a955-334ab24d276c-secret-metrics-server-client-certs\") pod \"metrics-server-597df9c44c-sqk79\" (UID: \"72b0bbb3-86be-47da-a955-334ab24d276c\") " pod="openshift-monitoring/metrics-server-597df9c44c-sqk79"
Sep 16 18:46:31.837952 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.837878    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/72b0bbb3-86be-47da-a955-334ab24d276c-client-ca-bundle\") pod \"metrics-server-597df9c44c-sqk79\" (UID: \"72b0bbb3-86be-47da-a955-334ab24d276c\") " pod="openshift-monitoring/metrics-server-597df9c44c-sqk79"
Sep 16 18:46:31.837952 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.837920    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/72b0bbb3-86be-47da-a955-334ab24d276c-secret-metrics-server-tls\") pod \"metrics-server-597df9c44c-sqk79\" (UID: \"72b0bbb3-86be-47da-a955-334ab24d276c\") " pod="openshift-monitoring/metrics-server-597df9c44c-sqk79"
Sep 16 18:46:31.837952 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.837948    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/72b0bbb3-86be-47da-a955-334ab24d276c-secret-metrics-server-client-certs\") pod \"metrics-server-597df9c44c-sqk79\" (UID: \"72b0bbb3-86be-47da-a955-334ab24d276c\") " pod="openshift-monitoring/metrics-server-597df9c44c-sqk79"
Sep 16 18:46:31.838185 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.838002    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/72b0bbb3-86be-47da-a955-334ab24d276c-metrics-server-audit-profiles\") pod \"metrics-server-597df9c44c-sqk79\" (UID: \"72b0bbb3-86be-47da-a955-334ab24d276c\") " pod="openshift-monitoring/metrics-server-597df9c44c-sqk79"
Sep 16 18:46:31.838185 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.838029    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6hd4\" (UniqueName: \"kubernetes.io/projected/72b0bbb3-86be-47da-a955-334ab24d276c-kube-api-access-j6hd4\") pod \"metrics-server-597df9c44c-sqk79\" (UID: \"72b0bbb3-86be-47da-a955-334ab24d276c\") " pod="openshift-monitoring/metrics-server-597df9c44c-sqk79"
Sep 16 18:46:31.838185 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.838059    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/72b0bbb3-86be-47da-a955-334ab24d276c-audit-log\") pod \"metrics-server-597df9c44c-sqk79\" (UID: \"72b0bbb3-86be-47da-a955-334ab24d276c\") " pod="openshift-monitoring/metrics-server-597df9c44c-sqk79"
Sep 16 18:46:31.838185 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.838097    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72b0bbb3-86be-47da-a955-334ab24d276c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-597df9c44c-sqk79\" (UID: \"72b0bbb3-86be-47da-a955-334ab24d276c\") " pod="openshift-monitoring/metrics-server-597df9c44c-sqk79"
Sep 16 18:46:31.838786 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.838714    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/72b0bbb3-86be-47da-a955-334ab24d276c-audit-log\") pod \"metrics-server-597df9c44c-sqk79\" (UID: \"72b0bbb3-86be-47da-a955-334ab24d276c\") " pod="openshift-monitoring/metrics-server-597df9c44c-sqk79"
Sep 16 18:46:31.838936 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.838910    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/72b0bbb3-86be-47da-a955-334ab24d276c-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-597df9c44c-sqk79\" (UID: \"72b0bbb3-86be-47da-a955-334ab24d276c\") " pod="openshift-monitoring/metrics-server-597df9c44c-sqk79"
Sep 16 18:46:31.839376 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.839327    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/72b0bbb3-86be-47da-a955-334ab24d276c-metrics-server-audit-profiles\") pod \"metrics-server-597df9c44c-sqk79\" (UID: \"72b0bbb3-86be-47da-a955-334ab24d276c\") " pod="openshift-monitoring/metrics-server-597df9c44c-sqk79"
Sep 16 18:46:31.841078 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.841043    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/72b0bbb3-86be-47da-a955-334ab24d276c-client-ca-bundle\") pod \"metrics-server-597df9c44c-sqk79\" (UID: \"72b0bbb3-86be-47da-a955-334ab24d276c\") " pod="openshift-monitoring/metrics-server-597df9c44c-sqk79"
Sep 16 18:46:31.841671 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.841512    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/72b0bbb3-86be-47da-a955-334ab24d276c-secret-metrics-server-client-certs\") pod \"metrics-server-597df9c44c-sqk79\" (UID: \"72b0bbb3-86be-47da-a955-334ab24d276c\") " pod="openshift-monitoring/metrics-server-597df9c44c-sqk79"
Sep 16 18:46:31.842080 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.842055    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/72b0bbb3-86be-47da-a955-334ab24d276c-secret-metrics-server-tls\") pod \"metrics-server-597df9c44c-sqk79\" (UID: \"72b0bbb3-86be-47da-a955-334ab24d276c\") " pod="openshift-monitoring/metrics-server-597df9c44c-sqk79"
Sep 16 18:46:31.852393 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.852375    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6hd4\" (UniqueName: \"kubernetes.io/projected/72b0bbb3-86be-47da-a955-334ab24d276c-kube-api-access-j6hd4\") pod \"metrics-server-597df9c44c-sqk79\" (UID: \"72b0bbb3-86be-47da-a955-334ab24d276c\") " pod="openshift-monitoring/metrics-server-597df9c44c-sqk79"
Sep 16 18:46:31.884318 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.884287    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-5b9ms"]
Sep 16 18:46:31.895901 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.895875    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-597df9c44c-sqk79"
Sep 16 18:46:31.906302 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.906283    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tjwj4"
Sep 16 18:46:31.906393 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.906321    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-s5zl7" event={"ID":"6239a312-62d8-4ea0-b52a-d96f7a4d233c","Type":"ContainerStarted","Data":"24d359dd4daca88681fa26de13c428a1c84389c5607057578a6417f88eb61ce1"}
Sep 16 18:46:31.906393 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.906360    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-5b9ms"]
Sep 16 18:46:31.906393 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.906384    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-s5zl7" event={"ID":"6239a312-62d8-4ea0-b52a-d96f7a4d233c","Type":"ContainerStarted","Data":"d024f091c8d2b196cf70e637609ea31e1407f6cae9a71c88b99ce20001c21251"}
Sep 16 18:46:31.906551 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.906409    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tjwj4" event={"ID":"d0bc4f19-10ce-472d-b31b-dd58bb9d7afa","Type":"ContainerStarted","Data":"1cb3e150f000463933a6b36e3da70c5001c9ca6928d1f1048e3d067858927ace"}
Sep 16 18:46:31.906551 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.906432    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tjwj4" event={"ID":"d0bc4f19-10ce-472d-b31b-dd58bb9d7afa","Type":"ContainerStarted","Data":"e429484f270f360b95cd6cc5c22223b950cf50c6ecb73e1f740191a2f9665865"}
Sep 16 18:46:31.906551 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.906505    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5b9ms"
Sep 16 18:46:31.908926 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.908905    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 16 18:46:31.909410 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.909394    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-dnq9g\""
Sep 16 18:46:31.933035 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.932968    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-s5zl7" podStartSLOduration=2.692714826 podStartE2EDuration="4.932954559s" totalImagesPullingTime="2.240239733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:31.931158926 +0000 UTC m=+102.006928559" watchObservedRunningTime="2026-09-16 18:46:31.932954559 +0000 UTC m=+102.008724184"
Sep 16 18:46:31.949669 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:31.949630    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tjwj4" podStartSLOduration=65.792897095 podStartE2EDuration="1m7.949617481s" totalImagesPullingTime="2.156720386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:31.949554879 +0000 UTC m=+102.025324513" watchObservedRunningTime="2026-09-16 18:46:31.949617481 +0000 UTC m=+102.025387110"
Sep 16 18:46:32.040037 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:32.040014    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/43f2ae30-2608-438b-8be9-e87438990203-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-5b9ms\" (UID: \"43f2ae30-2608-438b-8be9-e87438990203\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5b9ms"
Sep 16 18:46:32.040400 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:32.040163    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tpqd\" (UniqueName: \"kubernetes.io/projected/43f2ae30-2608-438b-8be9-e87438990203-kube-api-access-8tpqd\") pod \"monitoring-plugin-5d85778b9d-5b9ms\" (UID: \"43f2ae30-2608-438b-8be9-e87438990203\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5b9ms"
Sep 16 18:46:32.141613 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:32.141584    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tpqd\" (UniqueName: \"kubernetes.io/projected/43f2ae30-2608-438b-8be9-e87438990203-kube-api-access-8tpqd\") pod \"monitoring-plugin-5d85778b9d-5b9ms\" (UID: \"43f2ae30-2608-438b-8be9-e87438990203\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5b9ms"
Sep 16 18:46:32.141701 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:32.141667    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/43f2ae30-2608-438b-8be9-e87438990203-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-5b9ms\" (UID: \"43f2ae30-2608-438b-8be9-e87438990203\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5b9ms"
Sep 16 18:46:32.141835 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:46:32.141816    2554 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Sep 16 18:46:32.141930 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:46:32.141884    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43f2ae30-2608-438b-8be9-e87438990203-monitoring-plugin-cert podName:43f2ae30-2608-438b-8be9-e87438990203 nodeName:}" failed. No retries permitted until 2026-09-16 18:46:32.641865061 +0000 UTC m=+102.717634688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/43f2ae30-2608-438b-8be9-e87438990203-monitoring-plugin-cert") pod "monitoring-plugin-5d85778b9d-5b9ms" (UID: "43f2ae30-2608-438b-8be9-e87438990203") : secret "monitoring-plugin-cert" not found
Sep 16 18:46:32.153182 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:32.153162    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tpqd\" (UniqueName: \"kubernetes.io/projected/43f2ae30-2608-438b-8be9-e87438990203-kube-api-access-8tpqd\") pod \"monitoring-plugin-5d85778b9d-5b9ms\" (UID: \"43f2ae30-2608-438b-8be9-e87438990203\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5b9ms"
Sep 16 18:46:32.327270 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:32.327241    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-597df9c44c-sqk79"]
Sep 16 18:46:32.330478 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:46:32.330454    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b0bbb3_86be_47da_a955_334ab24d276c.slice/crio-e5e66cd352f10c3935fea2787ffba03d1b4cd2ac3e3f5878fc767d3238568584 WatchSource:0}: Error finding container e5e66cd352f10c3935fea2787ffba03d1b4cd2ac3e3f5878fc767d3238568584: Status 404 returned error can't find the container with id e5e66cd352f10c3935fea2787ffba03d1b4cd2ac3e3f5878fc767d3238568584
Sep 16 18:46:32.646428 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:32.646395    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/43f2ae30-2608-438b-8be9-e87438990203-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-5b9ms\" (UID: \"43f2ae30-2608-438b-8be9-e87438990203\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5b9ms"
Sep 16 18:46:32.649002 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:32.648975    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/43f2ae30-2608-438b-8be9-e87438990203-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-5b9ms\" (UID: \"43f2ae30-2608-438b-8be9-e87438990203\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5b9ms"
Sep 16 18:46:32.772687 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:32.772652    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dvtnm"
Sep 16 18:46:32.817515 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:32.817481    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5b9ms"
Sep 16 18:46:32.903943 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:32.903765    2554 generic.go:356] "Generic (PLEG): container finished" podID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerID="98140a5d9415b0f70c10360dec70dce7e7aab74e4625bec8ba8510ec757e9801" exitCode=0
Sep 16 18:46:32.903943 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:32.903827    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3adb4633-501b-4a6d-9a15-9ecbe0068e9f","Type":"ContainerDied","Data":"98140a5d9415b0f70c10360dec70dce7e7aab74e4625bec8ba8510ec757e9801"}
Sep 16 18:46:32.905503 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:32.905478    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-597df9c44c-sqk79" event={"ID":"72b0bbb3-86be-47da-a955-334ab24d276c","Type":"ContainerStarted","Data":"e5e66cd352f10c3935fea2787ffba03d1b4cd2ac3e3f5878fc767d3238568584"}
Sep 16 18:46:33.280450 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.280425    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-5b9ms"]
Sep 16 18:46:33.282529 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:46:33.282498    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f2ae30_2608_438b_8be9_e87438990203.slice/crio-8205fc756bc1f3f55b067da747d0ec52bbeb57d5856cc9b8f10353a25ee9e1de WatchSource:0}: Error finding container 8205fc756bc1f3f55b067da747d0ec52bbeb57d5856cc9b8f10353a25ee9e1de: Status 404 returned error can't find the container with id 8205fc756bc1f3f55b067da747d0ec52bbeb57d5856cc9b8f10353a25ee9e1de
Sep 16 18:46:33.616300 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.614943    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:46:33.618996 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.618965    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.621798 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.621400    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 18:46:33.621798 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.621629    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 18:46:33.627734 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.623936    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8mq8tjp6d6n9u\""
Sep 16 18:46:33.627734 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.624306    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 18:46:33.627734 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.624654    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 18:46:33.627734 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.624925    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 18:46:33.627734 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.625157    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 18:46:33.627734 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.625434    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 18:46:33.627734 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.625640    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 18:46:33.627734 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.625866    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 18:46:33.627734 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.626044    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fhlkj\""
Sep 16 18:46:33.627734 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.626220    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 18:46:33.628577 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.628553    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 18:46:33.629588 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.629565    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 18:46:33.629705 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.629569    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 18:46:33.630103 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.630080    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 18:46:33.645240 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.645211    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:46:33.755741 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.755692    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c209e205-9e15-43c5-a592-98a5d0d7ad21-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.755925 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.755754    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.755925 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.755784    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.755925 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.755832    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.755925 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.755910    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.756129 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.755944    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.756129 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.755976    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.756129 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.756023    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.756129 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.756046    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.756129 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.756075    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c209e205-9e15-43c5-a592-98a5d0d7ad21-config-out\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.756129 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.756121    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.756306 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.756155    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.756306 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.756180    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.756306 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.756209    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.756306 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.756239    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.756306 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.756272    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-web-config\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.756306 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.756299    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-config\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.756508 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.756325    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.756508 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.756352    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj9k4\" (UniqueName: \"kubernetes.io/projected/c209e205-9e15-43c5-a592-98a5d0d7ad21-kube-api-access-qj9k4\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.756508 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.756392    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.857828 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.857784    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.858278 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.857837    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.858278 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.857862    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.858278 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.858089    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.858278 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.858140    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.858278 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.858178    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-web-config\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.858278 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.858208    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-config\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.858278 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.858232    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.858278 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.858261    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj9k4\" (UniqueName: \"kubernetes.io/projected/c209e205-9e15-43c5-a592-98a5d0d7ad21-kube-api-access-qj9k4\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.858697 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.858300    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.858697 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.858342    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c209e205-9e15-43c5-a592-98a5d0d7ad21-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.858697 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.858368    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.858697 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.858392    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.858697 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.858441    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.858697 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.858469    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.858697 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.858495    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.858697 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.858522    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.858697 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.858567    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.858697 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.858587    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.858697 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.858615    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c209e205-9e15-43c5-a592-98a5d0d7ad21-config-out\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.859242 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.859096    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.861212 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.861171    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.861480 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.861457    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.862115 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.862091    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.863601 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.863498    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.864004 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.863979    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.864004 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.863990    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c209e205-9e15-43c5-a592-98a5d0d7ad21-config-out\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.864324 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.864249    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.865187 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.864747    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.865187 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.864948    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.865595 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.865572    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.865858 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.865835    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.866403 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.866335    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.867803 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.867756    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.867900 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.867842    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-config\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.868477 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.868438    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.868807 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.868785    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-web-config\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.869118 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.869078    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj9k4\" (UniqueName: \"kubernetes.io/projected/c209e205-9e15-43c5-a592-98a5d0d7ad21-kube-api-access-qj9k4\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.869297 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.869271    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c209e205-9e15-43c5-a592-98a5d0d7ad21-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.870021 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.869996    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:33.913452 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.913419    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5" event={"ID":"be8ef430-695c-4699-8ed3-46cfc365643e","Type":"ContainerStarted","Data":"1342a4cbd2bb7e5a869f0ed7e3c6b45517fdd72bd2c003d9ea98034efbcd9d93"}
Sep 16 18:46:33.913578 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.913504    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5" event={"ID":"be8ef430-695c-4699-8ed3-46cfc365643e","Type":"ContainerStarted","Data":"e7e3d9c52ccfa9b2b178bdc38fa946b28a8e1c80481b8133bdc60348d00891d9"}
Sep 16 18:46:33.913578 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.913550    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5" event={"ID":"be8ef430-695c-4699-8ed3-46cfc365643e","Type":"ContainerStarted","Data":"283617a037806f8fab39135e83d651cf9262c72ec0688c641d77dfbed0c37e4d"}
Sep 16 18:46:33.914956 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.914928    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5b9ms" event={"ID":"43f2ae30-2608-438b-8be9-e87438990203","Type":"ContainerStarted","Data":"8205fc756bc1f3f55b067da747d0ec52bbeb57d5856cc9b8f10353a25ee9e1de"}
Sep 16 18:46:33.937556 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:33.937535    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:35.177515 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:35.177372    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:46:35.182438 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:46:35.182283    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc209e205_9e15_43c5_a592_98a5d0d7ad21.slice/crio-73501e2a60d7c0262b95fc9dac7413bbeb9cc1c7e542958797baee986dcbcf4f WatchSource:0}: Error finding container 73501e2a60d7c0262b95fc9dac7413bbeb9cc1c7e542958797baee986dcbcf4f: Status 404 returned error can't find the container with id 73501e2a60d7c0262b95fc9dac7413bbeb9cc1c7e542958797baee986dcbcf4f
Sep 16 18:46:35.929057 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:35.929029    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-597df9c44c-sqk79" event={"ID":"72b0bbb3-86be-47da-a955-334ab24d276c","Type":"ContainerStarted","Data":"08d582e75840ead9d9e4ec240b3c39b04a1cdbb4cb240f7eb4a7a79893890467"}
Sep 16 18:46:35.932329 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:35.932291    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5" event={"ID":"be8ef430-695c-4699-8ed3-46cfc365643e","Type":"ContainerStarted","Data":"ebcb7c4d75971fe0c44b1a059f92dcaa3e5f8abeee83df4f3026680e5a5678d6"}
Sep 16 18:46:35.932473 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:35.932362    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5" event={"ID":"be8ef430-695c-4699-8ed3-46cfc365643e","Type":"ContainerStarted","Data":"a5d2650bd0d5f4bd2978a92c0faa0c5a09ad40d9dc5ccbcc9f56bd9f5ff08afd"}
Sep 16 18:46:35.932473 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:35.932409    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5" event={"ID":"be8ef430-695c-4699-8ed3-46cfc365643e","Type":"ContainerStarted","Data":"978d1fe060928caae109ff7156e03f3a237883145d31dfc09d798bdc273f11ee"}
Sep 16 18:46:35.932473 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:35.932452    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5"
Sep 16 18:46:35.933990 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:35.933968    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5b9ms" event={"ID":"43f2ae30-2608-438b-8be9-e87438990203","Type":"ContainerStarted","Data":"ba7a42ec3efbe50e8b63422390862777a0097b60ecc681636b5692dd945c0976"}
Sep 16 18:46:35.934135 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:35.934110    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5b9ms"
Sep 16 18:46:35.935790 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:35.935764    2554 generic.go:356] "Generic (PLEG): container finished" podID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerID="837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b" exitCode=0
Sep 16 18:46:35.936469 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:35.935927    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c209e205-9e15-43c5-a592-98a5d0d7ad21","Type":"ContainerDied","Data":"837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b"}
Sep 16 18:46:35.936469 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:35.935999    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c209e205-9e15-43c5-a592-98a5d0d7ad21","Type":"ContainerStarted","Data":"73501e2a60d7c0262b95fc9dac7413bbeb9cc1c7e542958797baee986dcbcf4f"}
Sep 16 18:46:35.940352 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:35.940335    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5b9ms"
Sep 16 18:46:35.940449 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:35.940370    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3adb4633-501b-4a6d-9a15-9ecbe0068e9f","Type":"ContainerStarted","Data":"41d9a48e381b0555c38a6d1a33ffba021e87def061da9545891cb453c8c6fe67"}
Sep 16 18:46:35.940449 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:35.940421    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3adb4633-501b-4a6d-9a15-9ecbe0068e9f","Type":"ContainerStarted","Data":"8176b0c1c152f963f7df51d5b5a6c6e5f611e400e4837635c340d0d6db8aaf0f"}
Sep 16 18:46:35.940564 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:35.940471    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3adb4633-501b-4a6d-9a15-9ecbe0068e9f","Type":"ContainerStarted","Data":"a3bc2a2d30ba2702998cfe656e741a79fb340c818607ba14462382392e096973"}
Sep 16 18:46:35.940564 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:35.940513    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3adb4633-501b-4a6d-9a15-9ecbe0068e9f","Type":"ContainerStarted","Data":"1e3a9a3107d4977efea948f16876e076d7faff463b8f8cbf4dea37b4a5df4882"}
Sep 16 18:46:35.940665 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:35.940561    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3adb4633-501b-4a6d-9a15-9ecbe0068e9f","Type":"ContainerStarted","Data":"a08a45cf05fcf2ba623851978cce6330e8e5387cc93f5eb49a38f77926ae0313"}
Sep 16 18:46:35.940665 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:35.940603    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3adb4633-501b-4a6d-9a15-9ecbe0068e9f","Type":"ContainerStarted","Data":"8a225a3ccb6947cff5863f2df1931c915610801c11c1e88af802e6489bb183a1"}
Sep 16 18:46:35.948418 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:35.948375    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-597df9c44c-sqk79" podStartSLOduration=2.394154263 podStartE2EDuration="4.948360574s" totalImagesPullingTime="2.554206311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:35.947277548 +0000 UTC m=+106.023047180" watchObservedRunningTime="2026-09-16 18:46:35.948360574 +0000 UTC m=+106.024130205"
Sep 16 18:46:36.007084 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:36.007042    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.757602835 podStartE2EDuration="8.007030099s" totalImagesPullingTime="4.238470099s" totalInitContainerRuntime="10.957165ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:36.005418504 +0000 UTC m=+106.081188170" watchObservedRunningTime="2026-09-16 18:46:36.007030099 +0000 UTC m=+106.082799730"
Sep 16 18:46:36.027820 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:36.027782    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-5b9ms" podStartSLOduration=3.422888731 podStartE2EDuration="5.027769501s" totalImagesPullingTime="1.60488077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:36.026415443 +0000 UTC m=+106.102185077" watchObservedRunningTime="2026-09-16 18:46:36.027769501 +0000 UTC m=+106.103539133"
Sep 16 18:46:36.089525 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:36.089473    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5" podStartSLOduration=3.068193839 podStartE2EDuration="7.08945512s" totalImagesPullingTime="4.021261281s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:36.087541548 +0000 UTC m=+106.163311180" watchObservedRunningTime="2026-09-16 18:46:36.08945512 +0000 UTC m=+106.165224753"
Sep 16 18:46:38.958489 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:38.958456    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c209e205-9e15-43c5-a592-98a5d0d7ad21","Type":"ContainerStarted","Data":"a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a"}
Sep 16 18:46:38.958793 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:38.958544    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c209e205-9e15-43c5-a592-98a5d0d7ad21","Type":"ContainerStarted","Data":"d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31"}
Sep 16 18:46:38.958793 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:38.958587    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c209e205-9e15-43c5-a592-98a5d0d7ad21","Type":"ContainerStarted","Data":"1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68"}
Sep 16 18:46:38.958793 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:38.958625    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c209e205-9e15-43c5-a592-98a5d0d7ad21","Type":"ContainerStarted","Data":"7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64"}
Sep 16 18:46:39.968480 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:39.968445    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c209e205-9e15-43c5-a592-98a5d0d7ad21","Type":"ContainerStarted","Data":"9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127"}
Sep 16 18:46:39.968834 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:39.968540    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c209e205-9e15-43c5-a592-98a5d0d7ad21","Type":"ContainerStarted","Data":"4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010"}
Sep 16 18:46:39.997390 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:39.997328    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.218517869 podStartE2EDuration="6.997310286s" totalImagesPullingTime="2.756391126s" totalInitContainerRuntime="22.401291ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:46:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:39.996403271 +0000 UTC m=+110.072172901" watchObservedRunningTime="2026-09-16 18:46:39.997310286 +0000 UTC m=+110.073079917"
Sep 16 18:46:40.275618 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.275578    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-558956f4c-ptzxv" podUID="2688504b-6527-42f6-a904-8ef74b146bc7" containerName="registry" containerID="cri-o://084aafd0b98d36d9fd0b97352ddf7bfd8e49ffebbcdfdcb4676e65e62cb53600" gracePeriod=30
Sep 16 18:46:40.441758 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.441734    2554 scope.go:117] "RemoveContainer" containerID="fe5696b3ea2ca860945e6aa76576bd547820abebb73e64bb2088dfc32c41b21e"
Sep 16 18:46:40.510540 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.510515    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:46:40.617839 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.617765    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2688504b-6527-42f6-a904-8ef74b146bc7-trusted-ca\") pod \"2688504b-6527-42f6-a904-8ef74b146bc7\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") "
Sep 16 18:46:40.617839 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.617797    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-bound-sa-token\") pod \"2688504b-6527-42f6-a904-8ef74b146bc7\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") "
Sep 16 18:46:40.617839 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.617817    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2688504b-6527-42f6-a904-8ef74b146bc7-installation-pull-secrets\") pod \"2688504b-6527-42f6-a904-8ef74b146bc7\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") "
Sep 16 18:46:40.618085 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.617950    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2688504b-6527-42f6-a904-8ef74b146bc7-registry-certificates\") pod \"2688504b-6527-42f6-a904-8ef74b146bc7\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") "
Sep 16 18:46:40.618085 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.617989    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2688504b-6527-42f6-a904-8ef74b146bc7-ca-trust-extracted\") pod \"2688504b-6527-42f6-a904-8ef74b146bc7\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") "
Sep 16 18:46:40.618085 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.618027    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2688504b-6527-42f6-a904-8ef74b146bc7-image-registry-private-configuration\") pod \"2688504b-6527-42f6-a904-8ef74b146bc7\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") "
Sep 16 18:46:40.618085 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.618074    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-registry-tls\") pod \"2688504b-6527-42f6-a904-8ef74b146bc7\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") "
Sep 16 18:46:40.618263 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.618113    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mp7j\" (UniqueName: \"kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-kube-api-access-8mp7j\") pod \"2688504b-6527-42f6-a904-8ef74b146bc7\" (UID: \"2688504b-6527-42f6-a904-8ef74b146bc7\") "
Sep 16 18:46:40.618316 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.618204    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2688504b-6527-42f6-a904-8ef74b146bc7-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2688504b-6527-42f6-a904-8ef74b146bc7" (UID: "2688504b-6527-42f6-a904-8ef74b146bc7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:40.618368 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.618310    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2688504b-6527-42f6-a904-8ef74b146bc7-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2688504b-6527-42f6-a904-8ef74b146bc7" (UID: "2688504b-6527-42f6-a904-8ef74b146bc7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:46:40.618469 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.618450    2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2688504b-6527-42f6-a904-8ef74b146bc7-trusted-ca\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:46:40.618521 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.618477    2554 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2688504b-6527-42f6-a904-8ef74b146bc7-registry-certificates\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:46:40.620711 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.620686    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-kube-api-access-8mp7j" (OuterVolumeSpecName: "kube-api-access-8mp7j") pod "2688504b-6527-42f6-a904-8ef74b146bc7" (UID: "2688504b-6527-42f6-a904-8ef74b146bc7"). InnerVolumeSpecName "kube-api-access-8mp7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:40.620711 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.620693    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2688504b-6527-42f6-a904-8ef74b146bc7-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2688504b-6527-42f6-a904-8ef74b146bc7" (UID: "2688504b-6527-42f6-a904-8ef74b146bc7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:40.620711 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.620737    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2688504b-6527-42f6-a904-8ef74b146bc7" (UID: "2688504b-6527-42f6-a904-8ef74b146bc7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:40.621020 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.620806    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2688504b-6527-42f6-a904-8ef74b146bc7-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2688504b-6527-42f6-a904-8ef74b146bc7" (UID: "2688504b-6527-42f6-a904-8ef74b146bc7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:46:40.621020 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.620846    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2688504b-6527-42f6-a904-8ef74b146bc7" (UID: "2688504b-6527-42f6-a904-8ef74b146bc7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:46:40.626138 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.626118    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2688504b-6527-42f6-a904-8ef74b146bc7-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2688504b-6527-42f6-a904-8ef74b146bc7" (UID: "2688504b-6527-42f6-a904-8ef74b146bc7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:46:40.719285 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.719262    2554 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2688504b-6527-42f6-a904-8ef74b146bc7-ca-trust-extracted\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:46:40.719285 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.719284    2554 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2688504b-6527-42f6-a904-8ef74b146bc7-image-registry-private-configuration\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:46:40.719404 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.719294    2554 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-registry-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:46:40.719404 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.719304    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mp7j\" (UniqueName: \"kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-kube-api-access-8mp7j\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:46:40.719404 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.719313    2554 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2688504b-6527-42f6-a904-8ef74b146bc7-bound-sa-token\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:46:40.719404 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.719321    2554 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2688504b-6527-42f6-a904-8ef74b146bc7-installation-pull-secrets\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:46:40.974955 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.974884    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/2.log"
Sep 16 18:46:40.975370 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.975004    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-696884fb9c-llwg5" event={"ID":"3589130e-56b7-4ce2-ad13-55804eabd1b3","Type":"ContainerStarted","Data":"1d92f8aeac01cea025a7f7df4eda7d4b9e7dcdeb965425e1030210486f35c086"}
Sep 16 18:46:40.975370 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.975329    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-696884fb9c-llwg5"
Sep 16 18:46:40.976424 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.976399    2554 generic.go:356] "Generic (PLEG): container finished" podID="2688504b-6527-42f6-a904-8ef74b146bc7" containerID="084aafd0b98d36d9fd0b97352ddf7bfd8e49ffebbcdfdcb4676e65e62cb53600" exitCode=0
Sep 16 18:46:40.976535 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.976444    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-558956f4c-ptzxv"
Sep 16 18:46:40.976535 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.976450    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-558956f4c-ptzxv" event={"ID":"2688504b-6527-42f6-a904-8ef74b146bc7","Type":"ContainerDied","Data":"084aafd0b98d36d9fd0b97352ddf7bfd8e49ffebbcdfdcb4676e65e62cb53600"}
Sep 16 18:46:40.976535 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.976494    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-558956f4c-ptzxv" event={"ID":"2688504b-6527-42f6-a904-8ef74b146bc7","Type":"ContainerDied","Data":"903bc1b8a23d49794a92046ee9b28282f09d36a351da24a099fb710229f30981"}
Sep 16 18:46:40.976535 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.976517    2554 scope.go:117] "RemoveContainer" containerID="084aafd0b98d36d9fd0b97352ddf7bfd8e49ffebbcdfdcb4676e65e62cb53600"
Sep 16 18:46:40.981049 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.981022    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-696884fb9c-llwg5"
Sep 16 18:46:40.984735 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.984621    2554 scope.go:117] "RemoveContainer" containerID="084aafd0b98d36d9fd0b97352ddf7bfd8e49ffebbcdfdcb4676e65e62cb53600"
Sep 16 18:46:40.984960 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:46:40.984933    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"084aafd0b98d36d9fd0b97352ddf7bfd8e49ffebbcdfdcb4676e65e62cb53600\": container with ID starting with 084aafd0b98d36d9fd0b97352ddf7bfd8e49ffebbcdfdcb4676e65e62cb53600 not found: ID does not exist" containerID="084aafd0b98d36d9fd0b97352ddf7bfd8e49ffebbcdfdcb4676e65e62cb53600"
Sep 16 18:46:40.985059 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.984968    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"084aafd0b98d36d9fd0b97352ddf7bfd8e49ffebbcdfdcb4676e65e62cb53600"} err="failed to get container status \"084aafd0b98d36d9fd0b97352ddf7bfd8e49ffebbcdfdcb4676e65e62cb53600\": rpc error: code = NotFound desc = could not find container \"084aafd0b98d36d9fd0b97352ddf7bfd8e49ffebbcdfdcb4676e65e62cb53600\": container with ID starting with 084aafd0b98d36d9fd0b97352ddf7bfd8e49ffebbcdfdcb4676e65e62cb53600 not found: ID does not exist"
Sep 16 18:46:40.994056 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:40.994018    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-696884fb9c-llwg5" podStartSLOduration=56.031718867 podStartE2EDuration="57.994006591s" totalImagesPullingTime="1.962287724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:45:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:46:40.99199413 +0000 UTC m=+111.067763761" watchObservedRunningTime="2026-09-16 18:46:40.994006591 +0000 UTC m=+111.069776240"
Sep 16 18:46:41.005026 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:41.005004    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-558956f4c-ptzxv"]
Sep 16 18:46:41.010009 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:41.009992    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-558956f4c-ptzxv"]
Sep 16 18:46:41.952201 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:41.952177    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-df7545c7c-qwct5"
Sep 16 18:46:42.442934 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:42.442900    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2688504b-6527-42f6-a904-8ef74b146bc7" path="/var/lib/kubelet/pods/2688504b-6527-42f6-a904-8ef74b146bc7/volumes"
Sep 16 18:46:42.917282 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:42.917258    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tjwj4"
Sep 16 18:46:43.938706 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:43.938671    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:46:51.896862 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:51.896828    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-597df9c44c-sqk79"
Sep 16 18:46:51.897211 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:51.896913    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-597df9c44c-sqk79"
Sep 16 18:46:59.484461 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:59.484436    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3adb4633-501b-4a6d-9a15-9ecbe0068e9f/init-config-reloader/0.log"
Sep 16 18:46:59.493805 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:59.493779    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3adb4633-501b-4a6d-9a15-9ecbe0068e9f/alertmanager/0.log"
Sep 16 18:46:59.643826 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:59.643799    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3adb4633-501b-4a6d-9a15-9ecbe0068e9f/config-reloader/0.log"
Sep 16 18:46:59.843810 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:46:59.843788    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3adb4633-501b-4a6d-9a15-9ecbe0068e9f/kube-rbac-proxy-web/0.log"
Sep 16 18:47:00.044585 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:00.044560    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3adb4633-501b-4a6d-9a15-9ecbe0068e9f/kube-rbac-proxy/0.log"
Sep 16 18:47:00.244165 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:00.244110    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3adb4633-501b-4a6d-9a15-9ecbe0068e9f/kube-rbac-proxy-metric/0.log"
Sep 16 18:47:00.268393 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:00.268371    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59dba60-7c0d-4e12-b53a-3462d4b9791a-metrics-certs\") pod \"network-metrics-daemon-zp95h\" (UID: \"d59dba60-7c0d-4e12-b53a-3462d4b9791a\") " pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:47:00.270706 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:00.270687    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d59dba60-7c0d-4e12-b53a-3462d4b9791a-metrics-certs\") pod \"network-metrics-daemon-zp95h\" (UID: \"d59dba60-7c0d-4e12-b53a-3462d4b9791a\") " pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:47:00.353908 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:00.353887    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-rlx6q\""
Sep 16 18:47:00.360744 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:00.360726    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zp95h"
Sep 16 18:47:00.443451 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:00.443426    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3adb4633-501b-4a6d-9a15-9ecbe0068e9f/prom-label-proxy/0.log"
Sep 16 18:47:00.636377 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:00.636352    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zp95h"]
Sep 16 18:47:00.639009 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:47:00.638987    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59dba60_7c0d_4e12_b53a_3462d4b9791a.slice/crio-68e001cb704284ada4a6d4b5fb591ba9502090baf9eb8efddbbca24256817b0c WatchSource:0}: Error finding container 68e001cb704284ada4a6d4b5fb591ba9502090baf9eb8efddbbca24256817b0c: Status 404 returned error can't find the container with id 68e001cb704284ada4a6d4b5fb591ba9502090baf9eb8efddbbca24256817b0c
Sep 16 18:47:00.644442 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:00.644388    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-ln8nz_b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5/cluster-monitoring-operator/0.log"
Sep 16 18:47:00.844129 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:00.844102    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-s5zl7_6239a312-62d8-4ea0-b52a-d96f7a4d233c/kube-state-metrics/0.log"
Sep 16 18:47:01.044632 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:01.044606    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-s5zl7_6239a312-62d8-4ea0-b52a-d96f7a4d233c/kube-rbac-proxy-main/0.log"
Sep 16 18:47:01.052276 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:01.052250    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zp95h" event={"ID":"d59dba60-7c0d-4e12-b53a-3462d4b9791a","Type":"ContainerStarted","Data":"68e001cb704284ada4a6d4b5fb591ba9502090baf9eb8efddbbca24256817b0c"}
Sep 16 18:47:01.243239 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:01.243192    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-s5zl7_6239a312-62d8-4ea0-b52a-d96f7a4d233c/kube-rbac-proxy-self/0.log"
Sep 16 18:47:01.443496 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:01.443425    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-597df9c44c-sqk79_72b0bbb3-86be-47da-a955-334ab24d276c/metrics-server/0.log"
Sep 16 18:47:01.643728 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:01.643693    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-5b9ms_43f2ae30-2608-438b-8be9-e87438990203/monitoring-plugin/0.log"
Sep 16 18:47:02.057806 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:02.057778    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zp95h" event={"ID":"d59dba60-7c0d-4e12-b53a-3462d4b9791a","Type":"ContainerStarted","Data":"987829cbf6524574304e0f71885bc238050ae342885626353bca9db946726c24"}
Sep 16 18:47:02.057966 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:02.057827    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zp95h" event={"ID":"d59dba60-7c0d-4e12-b53a-3462d4b9791a","Type":"ContainerStarted","Data":"9108fb2a6a20244bf036891441c8f4bde783f4c3dce3bfbb2bcdc29e84727f4b"}
Sep 16 18:47:02.082253 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:02.082211    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zp95h" podStartSLOduration=130.181677269 podStartE2EDuration="2m11.082197411s" totalImagesPullingTime="900.520142ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:44:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:02.079538611 +0000 UTC m=+132.155308243" watchObservedRunningTime="2026-09-16 18:47:02.082197411 +0000 UTC m=+132.157967042"
Sep 16 18:47:03.043336 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:03.043306    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9z9fb_c745eb77-57e1-4567-b68f-01e301940170/init-textfile/0.log"
Sep 16 18:47:03.244285 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:03.244251    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9z9fb_c745eb77-57e1-4567-b68f-01e301940170/node-exporter/0.log"
Sep 16 18:47:03.444030 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:03.443972    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9z9fb_c745eb77-57e1-4567-b68f-01e301940170/kube-rbac-proxy/0.log"
Sep 16 18:47:04.244422 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:04.244400    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c209e205-9e15-43c5-a592-98a5d0d7ad21/init-config-reloader/0.log"
Sep 16 18:47:04.445578 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:04.445553    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c209e205-9e15-43c5-a592-98a5d0d7ad21/prometheus/0.log"
Sep 16 18:47:04.644017 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:04.643993    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c209e205-9e15-43c5-a592-98a5d0d7ad21/config-reloader/0.log"
Sep 16 18:47:04.843938 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:04.843914    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c209e205-9e15-43c5-a592-98a5d0d7ad21/thanos-sidecar/0.log"
Sep 16 18:47:05.052546 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:05.052526    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c209e205-9e15-43c5-a592-98a5d0d7ad21/kube-rbac-proxy-web/0.log"
Sep 16 18:47:05.243321 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:05.243300    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c209e205-9e15-43c5-a592-98a5d0d7ad21/kube-rbac-proxy/0.log"
Sep 16 18:47:05.444467 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:05.444417    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_c209e205-9e15-43c5-a592-98a5d0d7ad21/kube-rbac-proxy-thanos/0.log"
Sep 16 18:47:06.044221 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:06.044198    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-zn2th_b480ee02-4a14-4111-84f6-21920933edba/prometheus-operator-admission-webhook/0.log"
Sep 16 18:47:06.243437 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:06.243415    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-df7545c7c-qwct5_be8ef430-695c-4699-8ed3-46cfc365643e/thanos-query/0.log"
Sep 16 18:47:06.445240 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:06.445181    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-df7545c7c-qwct5_be8ef430-695c-4699-8ed3-46cfc365643e/kube-rbac-proxy-web/0.log"
Sep 16 18:47:06.646505 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:06.646484    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-df7545c7c-qwct5_be8ef430-695c-4699-8ed3-46cfc365643e/kube-rbac-proxy/0.log"
Sep 16 18:47:06.847133 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:06.847108    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-df7545c7c-qwct5_be8ef430-695c-4699-8ed3-46cfc365643e/prom-label-proxy/0.log"
Sep 16 18:47:07.046066 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:07.046040    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-df7545c7c-qwct5_be8ef430-695c-4699-8ed3-46cfc365643e/kube-rbac-proxy-rules/0.log"
Sep 16 18:47:07.247483 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:07.247418    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-df7545c7c-qwct5_be8ef430-695c-4699-8ed3-46cfc365643e/kube-rbac-proxy-metrics/0.log"
Sep 16 18:47:07.644557 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:07.644536    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/2.log"
Sep 16 18:47:07.845330 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:07.845303    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/3.log"
Sep 16 18:47:11.904324 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:11.904283    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-597df9c44c-sqk79"
Sep 16 18:47:11.910905 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:11.910876    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-597df9c44c-sqk79"
Sep 16 18:47:13.101661 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:13.101628    2554 generic.go:356] "Generic (PLEG): container finished" podID="902005e5-c11d-408c-a3f6-82482c7b4e45" containerID="6a6632232aaf33cec28404734832bb646109cbeb31bc226a061530f790000733" exitCode=0
Sep 16 18:47:13.102035 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:13.101697    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5qjn2" event={"ID":"902005e5-c11d-408c-a3f6-82482c7b4e45","Type":"ContainerDied","Data":"6a6632232aaf33cec28404734832bb646109cbeb31bc226a061530f790000733"}
Sep 16 18:47:13.102081 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:13.102069    2554 scope.go:117] "RemoveContainer" containerID="6a6632232aaf33cec28404734832bb646109cbeb31bc226a061530f790000733"
Sep 16 18:47:14.107784 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:14.107751    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-5qjn2" event={"ID":"902005e5-c11d-408c-a3f6-82482c7b4e45","Type":"ContainerStarted","Data":"d0bdb9e14c73a643015c2baedc07635981d7a12af8223882a574092ca184f912"}
Sep 16 18:47:33.938081 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:33.938031    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:33.957488 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:33.957466    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:34.198782 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:34.198678    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:47.917042 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:47.916963    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:47:47.917499 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:47.917282    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerName="alertmanager" containerID="cri-o://8a225a3ccb6947cff5863f2df1931c915610801c11c1e88af802e6489bb183a1" gracePeriod=120
Sep 16 18:47:47.917499 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:47.917343    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerName="kube-rbac-proxy-metric" containerID="cri-o://8176b0c1c152f963f7df51d5b5a6c6e5f611e400e4837635c340d0d6db8aaf0f" gracePeriod=120
Sep 16 18:47:47.917499 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:47.917360    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerName="kube-rbac-proxy-web" containerID="cri-o://1e3a9a3107d4977efea948f16876e076d7faff463b8f8cbf4dea37b4a5df4882" gracePeriod=120
Sep 16 18:47:47.917499 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:47.917391    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerName="prom-label-proxy" containerID="cri-o://41d9a48e381b0555c38a6d1a33ffba021e87def061da9545891cb453c8c6fe67" gracePeriod=120
Sep 16 18:47:47.917499 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:47.917399    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerName="config-reloader" containerID="cri-o://a08a45cf05fcf2ba623851978cce6330e8e5387cc93f5eb49a38f77926ae0313" gracePeriod=120
Sep 16 18:47:47.917499 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:47.917370    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerName="kube-rbac-proxy" containerID="cri-o://a3bc2a2d30ba2702998cfe656e741a79fb340c818607ba14462382392e096973" gracePeriod=120
Sep 16 18:47:48.243058 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:48.242971    2554 generic.go:356] "Generic (PLEG): container finished" podID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerID="41d9a48e381b0555c38a6d1a33ffba021e87def061da9545891cb453c8c6fe67" exitCode=0
Sep 16 18:47:48.243058 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:48.242995    2554 generic.go:356] "Generic (PLEG): container finished" podID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerID="a3bc2a2d30ba2702998cfe656e741a79fb340c818607ba14462382392e096973" exitCode=0
Sep 16 18:47:48.243058 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:48.243002    2554 generic.go:356] "Generic (PLEG): container finished" podID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerID="a08a45cf05fcf2ba623851978cce6330e8e5387cc93f5eb49a38f77926ae0313" exitCode=0
Sep 16 18:47:48.243058 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:48.243008    2554 generic.go:356] "Generic (PLEG): container finished" podID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerID="8a225a3ccb6947cff5863f2df1931c915610801c11c1e88af802e6489bb183a1" exitCode=0
Sep 16 18:47:48.243058 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:48.243039    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3adb4633-501b-4a6d-9a15-9ecbe0068e9f","Type":"ContainerDied","Data":"41d9a48e381b0555c38a6d1a33ffba021e87def061da9545891cb453c8c6fe67"}
Sep 16 18:47:48.243363 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:48.243120    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3adb4633-501b-4a6d-9a15-9ecbe0068e9f","Type":"ContainerDied","Data":"a3bc2a2d30ba2702998cfe656e741a79fb340c818607ba14462382392e096973"}
Sep 16 18:47:48.243363 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:48.243160    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3adb4633-501b-4a6d-9a15-9ecbe0068e9f","Type":"ContainerDied","Data":"a08a45cf05fcf2ba623851978cce6330e8e5387cc93f5eb49a38f77926ae0313"}
Sep 16 18:47:48.243363 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:48.243207    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3adb4633-501b-4a6d-9a15-9ecbe0068e9f","Type":"ContainerDied","Data":"8a225a3ccb6947cff5863f2df1931c915610801c11c1e88af802e6489bb183a1"}
Sep 16 18:47:49.156241 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.156219    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.250210 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.250179    2554 generic.go:356] "Generic (PLEG): container finished" podID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerID="8176b0c1c152f963f7df51d5b5a6c6e5f611e400e4837635c340d0d6db8aaf0f" exitCode=0
Sep 16 18:47:49.250210 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.250207    2554 generic.go:356] "Generic (PLEG): container finished" podID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerID="1e3a9a3107d4977efea948f16876e076d7faff463b8f8cbf4dea37b4a5df4882" exitCode=0
Sep 16 18:47:49.250385 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.250252    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.250385 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.250255    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3adb4633-501b-4a6d-9a15-9ecbe0068e9f","Type":"ContainerDied","Data":"8176b0c1c152f963f7df51d5b5a6c6e5f611e400e4837635c340d0d6db8aaf0f"}
Sep 16 18:47:49.250385 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.250329    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3adb4633-501b-4a6d-9a15-9ecbe0068e9f","Type":"ContainerDied","Data":"1e3a9a3107d4977efea948f16876e076d7faff463b8f8cbf4dea37b4a5df4882"}
Sep 16 18:47:49.250385 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.250357    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3adb4633-501b-4a6d-9a15-9ecbe0068e9f","Type":"ContainerDied","Data":"c707c2f3403c35054002826ccfbd92ed9d2042c06d11254e65d8553c58a63a7c"}
Sep 16 18:47:49.250515 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.250396    2554 scope.go:117] "RemoveContainer" containerID="41d9a48e381b0555c38a6d1a33ffba021e87def061da9545891cb453c8c6fe67"
Sep 16 18:47:49.257913 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.257896    2554 scope.go:117] "RemoveContainer" containerID="8176b0c1c152f963f7df51d5b5a6c6e5f611e400e4837635c340d0d6db8aaf0f"
Sep 16 18:47:49.264897 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.264881    2554 scope.go:117] "RemoveContainer" containerID="a3bc2a2d30ba2702998cfe656e741a79fb340c818607ba14462382392e096973"
Sep 16 18:47:49.271300 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.271286    2554 scope.go:117] "RemoveContainer" containerID="1e3a9a3107d4977efea948f16876e076d7faff463b8f8cbf4dea37b4a5df4882"
Sep 16 18:47:49.277479 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.277461    2554 scope.go:117] "RemoveContainer" containerID="a08a45cf05fcf2ba623851978cce6330e8e5387cc93f5eb49a38f77926ae0313"
Sep 16 18:47:49.283992 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.283978    2554 scope.go:117] "RemoveContainer" containerID="8a225a3ccb6947cff5863f2df1931c915610801c11c1e88af802e6489bb183a1"
Sep 16 18:47:49.291862 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.291820    2554 scope.go:117] "RemoveContainer" containerID="98140a5d9415b0f70c10360dec70dce7e7aab74e4625bec8ba8510ec757e9801"
Sep 16 18:47:49.292107 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.292086    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-secret-alertmanager-kube-rbac-proxy\") pod \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") "
Sep 16 18:47:49.292200 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.292131    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-tls-assets\") pod \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") "
Sep 16 18:47:49.292200 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.292157    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-cluster-tls-config\") pod \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") "
Sep 16 18:47:49.292200 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.292187    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-metrics-client-ca\") pod \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") "
Sep 16 18:47:49.292334 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.292230    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-config-out\") pod \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") "
Sep 16 18:47:49.292334 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.292255    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-alertmanager-main-db\") pod \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") "
Sep 16 18:47:49.292334 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.292276    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-alertmanager-trusted-ca-bundle\") pod \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") "
Sep 16 18:47:49.292334 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.292296    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-config-volume\") pod \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") "
Sep 16 18:47:49.292514 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.292349    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrrrz\" (UniqueName: \"kubernetes.io/projected/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-kube-api-access-vrrrz\") pod \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") "
Sep 16 18:47:49.292514 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.292366    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-secret-alertmanager-kube-rbac-proxy-web\") pod \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") "
Sep 16 18:47:49.292514 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.292389    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-web-config\") pod \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") "
Sep 16 18:47:49.292514 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.292418    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") "
Sep 16 18:47:49.292514 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.292460    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-secret-alertmanager-main-tls\") pod \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\" (UID: \"3adb4633-501b-4a6d-9a15-9ecbe0068e9f\") "
Sep 16 18:47:49.292812 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.292781    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "3adb4633-501b-4a6d-9a15-9ecbe0068e9f" (UID: "3adb4633-501b-4a6d-9a15-9ecbe0068e9f"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:49.293459 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.293154    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "3adb4633-501b-4a6d-9a15-9ecbe0068e9f" (UID: "3adb4633-501b-4a6d-9a15-9ecbe0068e9f"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:49.293911 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.293875    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "3adb4633-501b-4a6d-9a15-9ecbe0068e9f" (UID: "3adb4633-501b-4a6d-9a15-9ecbe0068e9f"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:49.296546 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.296510    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "3adb4633-501b-4a6d-9a15-9ecbe0068e9f" (UID: "3adb4633-501b-4a6d-9a15-9ecbe0068e9f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:49.296673 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.296648    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "3adb4633-501b-4a6d-9a15-9ecbe0068e9f" (UID: "3adb4633-501b-4a6d-9a15-9ecbe0068e9f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:49.296673 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.296660    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "3adb4633-501b-4a6d-9a15-9ecbe0068e9f" (UID: "3adb4633-501b-4a6d-9a15-9ecbe0068e9f"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:49.296819 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.296774    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-config-volume" (OuterVolumeSpecName: "config-volume") pod "3adb4633-501b-4a6d-9a15-9ecbe0068e9f" (UID: "3adb4633-501b-4a6d-9a15-9ecbe0068e9f"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:49.296910 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.296885    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "3adb4633-501b-4a6d-9a15-9ecbe0068e9f" (UID: "3adb4633-501b-4a6d-9a15-9ecbe0068e9f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:49.297076 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.297044    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-config-out" (OuterVolumeSpecName: "config-out") pod "3adb4633-501b-4a6d-9a15-9ecbe0068e9f" (UID: "3adb4633-501b-4a6d-9a15-9ecbe0068e9f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:49.297159 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.297136    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-kube-api-access-vrrrz" (OuterVolumeSpecName: "kube-api-access-vrrrz") pod "3adb4633-501b-4a6d-9a15-9ecbe0068e9f" (UID: "3adb4633-501b-4a6d-9a15-9ecbe0068e9f"). InnerVolumeSpecName "kube-api-access-vrrrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:49.297337 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.297318    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "3adb4633-501b-4a6d-9a15-9ecbe0068e9f" (UID: "3adb4633-501b-4a6d-9a15-9ecbe0068e9f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:49.300289 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.300195    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "3adb4633-501b-4a6d-9a15-9ecbe0068e9f" (UID: "3adb4633-501b-4a6d-9a15-9ecbe0068e9f"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:49.306907 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.306877    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-web-config" (OuterVolumeSpecName: "web-config") pod "3adb4633-501b-4a6d-9a15-9ecbe0068e9f" (UID: "3adb4633-501b-4a6d-9a15-9ecbe0068e9f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:49.308303 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.308284    2554 scope.go:117] "RemoveContainer" containerID="41d9a48e381b0555c38a6d1a33ffba021e87def061da9545891cb453c8c6fe67"
Sep 16 18:47:49.308569 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:47:49.308551    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41d9a48e381b0555c38a6d1a33ffba021e87def061da9545891cb453c8c6fe67\": container with ID starting with 41d9a48e381b0555c38a6d1a33ffba021e87def061da9545891cb453c8c6fe67 not found: ID does not exist" containerID="41d9a48e381b0555c38a6d1a33ffba021e87def061da9545891cb453c8c6fe67"
Sep 16 18:47:49.308614 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.308580    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d9a48e381b0555c38a6d1a33ffba021e87def061da9545891cb453c8c6fe67"} err="failed to get container status \"41d9a48e381b0555c38a6d1a33ffba021e87def061da9545891cb453c8c6fe67\": rpc error: code = NotFound desc = could not find container \"41d9a48e381b0555c38a6d1a33ffba021e87def061da9545891cb453c8c6fe67\": container with ID starting with 41d9a48e381b0555c38a6d1a33ffba021e87def061da9545891cb453c8c6fe67 not found: ID does not exist"
Sep 16 18:47:49.308614 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.308598    2554 scope.go:117] "RemoveContainer" containerID="8176b0c1c152f963f7df51d5b5a6c6e5f611e400e4837635c340d0d6db8aaf0f"
Sep 16 18:47:49.308864 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:47:49.308845    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8176b0c1c152f963f7df51d5b5a6c6e5f611e400e4837635c340d0d6db8aaf0f\": container with ID starting with 8176b0c1c152f963f7df51d5b5a6c6e5f611e400e4837635c340d0d6db8aaf0f not found: ID does not exist" containerID="8176b0c1c152f963f7df51d5b5a6c6e5f611e400e4837635c340d0d6db8aaf0f"
Sep 16 18:47:49.308958 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.308872    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8176b0c1c152f963f7df51d5b5a6c6e5f611e400e4837635c340d0d6db8aaf0f"} err="failed to get container status \"8176b0c1c152f963f7df51d5b5a6c6e5f611e400e4837635c340d0d6db8aaf0f\": rpc error: code = NotFound desc = could not find container \"8176b0c1c152f963f7df51d5b5a6c6e5f611e400e4837635c340d0d6db8aaf0f\": container with ID starting with 8176b0c1c152f963f7df51d5b5a6c6e5f611e400e4837635c340d0d6db8aaf0f not found: ID does not exist"
Sep 16 18:47:49.308958 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.308893    2554 scope.go:117] "RemoveContainer" containerID="a3bc2a2d30ba2702998cfe656e741a79fb340c818607ba14462382392e096973"
Sep 16 18:47:49.309141 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:47:49.309124    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3bc2a2d30ba2702998cfe656e741a79fb340c818607ba14462382392e096973\": container with ID starting with a3bc2a2d30ba2702998cfe656e741a79fb340c818607ba14462382392e096973 not found: ID does not exist" containerID="a3bc2a2d30ba2702998cfe656e741a79fb340c818607ba14462382392e096973"
Sep 16 18:47:49.309182 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.309145    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3bc2a2d30ba2702998cfe656e741a79fb340c818607ba14462382392e096973"} err="failed to get container status \"a3bc2a2d30ba2702998cfe656e741a79fb340c818607ba14462382392e096973\": rpc error: code = NotFound desc = could not find container \"a3bc2a2d30ba2702998cfe656e741a79fb340c818607ba14462382392e096973\": container with ID starting with a3bc2a2d30ba2702998cfe656e741a79fb340c818607ba14462382392e096973 not found: ID does not exist"
Sep 16 18:47:49.309182 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.309158    2554 scope.go:117] "RemoveContainer" containerID="1e3a9a3107d4977efea948f16876e076d7faff463b8f8cbf4dea37b4a5df4882"
Sep 16 18:47:49.309387 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:47:49.309372    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e3a9a3107d4977efea948f16876e076d7faff463b8f8cbf4dea37b4a5df4882\": container with ID starting with 1e3a9a3107d4977efea948f16876e076d7faff463b8f8cbf4dea37b4a5df4882 not found: ID does not exist" containerID="1e3a9a3107d4977efea948f16876e076d7faff463b8f8cbf4dea37b4a5df4882"
Sep 16 18:47:49.309450 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.309395    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e3a9a3107d4977efea948f16876e076d7faff463b8f8cbf4dea37b4a5df4882"} err="failed to get container status \"1e3a9a3107d4977efea948f16876e076d7faff463b8f8cbf4dea37b4a5df4882\": rpc error: code = NotFound desc = could not find container \"1e3a9a3107d4977efea948f16876e076d7faff463b8f8cbf4dea37b4a5df4882\": container with ID starting with 1e3a9a3107d4977efea948f16876e076d7faff463b8f8cbf4dea37b4a5df4882 not found: ID does not exist"
Sep 16 18:47:49.309450 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.309415    2554 scope.go:117] "RemoveContainer" containerID="a08a45cf05fcf2ba623851978cce6330e8e5387cc93f5eb49a38f77926ae0313"
Sep 16 18:47:49.309679 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:47:49.309663    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a08a45cf05fcf2ba623851978cce6330e8e5387cc93f5eb49a38f77926ae0313\": container with ID starting with a08a45cf05fcf2ba623851978cce6330e8e5387cc93f5eb49a38f77926ae0313 not found: ID does not exist" containerID="a08a45cf05fcf2ba623851978cce6330e8e5387cc93f5eb49a38f77926ae0313"
Sep 16 18:47:49.309763 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.309683    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a08a45cf05fcf2ba623851978cce6330e8e5387cc93f5eb49a38f77926ae0313"} err="failed to get container status \"a08a45cf05fcf2ba623851978cce6330e8e5387cc93f5eb49a38f77926ae0313\": rpc error: code = NotFound desc = could not find container \"a08a45cf05fcf2ba623851978cce6330e8e5387cc93f5eb49a38f77926ae0313\": container with ID starting with a08a45cf05fcf2ba623851978cce6330e8e5387cc93f5eb49a38f77926ae0313 not found: ID does not exist"
Sep 16 18:47:49.309763 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.309696    2554 scope.go:117] "RemoveContainer" containerID="8a225a3ccb6947cff5863f2df1931c915610801c11c1e88af802e6489bb183a1"
Sep 16 18:47:49.309965 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:47:49.309947    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a225a3ccb6947cff5863f2df1931c915610801c11c1e88af802e6489bb183a1\": container with ID starting with 8a225a3ccb6947cff5863f2df1931c915610801c11c1e88af802e6489bb183a1 not found: ID does not exist" containerID="8a225a3ccb6947cff5863f2df1931c915610801c11c1e88af802e6489bb183a1"
Sep 16 18:47:49.310007 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.309971    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a225a3ccb6947cff5863f2df1931c915610801c11c1e88af802e6489bb183a1"} err="failed to get container status \"8a225a3ccb6947cff5863f2df1931c915610801c11c1e88af802e6489bb183a1\": rpc error: code = NotFound desc = could not find container \"8a225a3ccb6947cff5863f2df1931c915610801c11c1e88af802e6489bb183a1\": container with ID starting with 8a225a3ccb6947cff5863f2df1931c915610801c11c1e88af802e6489bb183a1 not found: ID does not exist"
Sep 16 18:47:49.310007 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.309987    2554 scope.go:117] "RemoveContainer" containerID="98140a5d9415b0f70c10360dec70dce7e7aab74e4625bec8ba8510ec757e9801"
Sep 16 18:47:49.310212 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:47:49.310195    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98140a5d9415b0f70c10360dec70dce7e7aab74e4625bec8ba8510ec757e9801\": container with ID starting with 98140a5d9415b0f70c10360dec70dce7e7aab74e4625bec8ba8510ec757e9801 not found: ID does not exist" containerID="98140a5d9415b0f70c10360dec70dce7e7aab74e4625bec8ba8510ec757e9801"
Sep 16 18:47:49.310267 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.310217    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98140a5d9415b0f70c10360dec70dce7e7aab74e4625bec8ba8510ec757e9801"} err="failed to get container status \"98140a5d9415b0f70c10360dec70dce7e7aab74e4625bec8ba8510ec757e9801\": rpc error: code = NotFound desc = could not find container \"98140a5d9415b0f70c10360dec70dce7e7aab74e4625bec8ba8510ec757e9801\": container with ID starting with 98140a5d9415b0f70c10360dec70dce7e7aab74e4625bec8ba8510ec757e9801 not found: ID does not exist"
Sep 16 18:47:49.310267 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.310239    2554 scope.go:117] "RemoveContainer" containerID="41d9a48e381b0555c38a6d1a33ffba021e87def061da9545891cb453c8c6fe67"
Sep 16 18:47:49.310467 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.310450    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41d9a48e381b0555c38a6d1a33ffba021e87def061da9545891cb453c8c6fe67"} err="failed to get container status \"41d9a48e381b0555c38a6d1a33ffba021e87def061da9545891cb453c8c6fe67\": rpc error: code = NotFound desc = could not find container \"41d9a48e381b0555c38a6d1a33ffba021e87def061da9545891cb453c8c6fe67\": container with ID starting with 41d9a48e381b0555c38a6d1a33ffba021e87def061da9545891cb453c8c6fe67 not found: ID does not exist"
Sep 16 18:47:49.310546 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.310467    2554 scope.go:117] "RemoveContainer" containerID="8176b0c1c152f963f7df51d5b5a6c6e5f611e400e4837635c340d0d6db8aaf0f"
Sep 16 18:47:49.310683 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.310669    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8176b0c1c152f963f7df51d5b5a6c6e5f611e400e4837635c340d0d6db8aaf0f"} err="failed to get container status \"8176b0c1c152f963f7df51d5b5a6c6e5f611e400e4837635c340d0d6db8aaf0f\": rpc error: code = NotFound desc = could not find container \"8176b0c1c152f963f7df51d5b5a6c6e5f611e400e4837635c340d0d6db8aaf0f\": container with ID starting with 8176b0c1c152f963f7df51d5b5a6c6e5f611e400e4837635c340d0d6db8aaf0f not found: ID does not exist"
Sep 16 18:47:49.310683 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.310682    2554 scope.go:117] "RemoveContainer" containerID="a3bc2a2d30ba2702998cfe656e741a79fb340c818607ba14462382392e096973"
Sep 16 18:47:49.310916 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.310899    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3bc2a2d30ba2702998cfe656e741a79fb340c818607ba14462382392e096973"} err="failed to get container status \"a3bc2a2d30ba2702998cfe656e741a79fb340c818607ba14462382392e096973\": rpc error: code = NotFound desc = could not find container \"a3bc2a2d30ba2702998cfe656e741a79fb340c818607ba14462382392e096973\": container with ID starting with a3bc2a2d30ba2702998cfe656e741a79fb340c818607ba14462382392e096973 not found: ID does not exist"
Sep 16 18:47:49.310962 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.310916    2554 scope.go:117] "RemoveContainer" containerID="1e3a9a3107d4977efea948f16876e076d7faff463b8f8cbf4dea37b4a5df4882"
Sep 16 18:47:49.311159 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.311141    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e3a9a3107d4977efea948f16876e076d7faff463b8f8cbf4dea37b4a5df4882"} err="failed to get container status \"1e3a9a3107d4977efea948f16876e076d7faff463b8f8cbf4dea37b4a5df4882\": rpc error: code = NotFound desc = could not find container \"1e3a9a3107d4977efea948f16876e076d7faff463b8f8cbf4dea37b4a5df4882\": container with ID starting with 1e3a9a3107d4977efea948f16876e076d7faff463b8f8cbf4dea37b4a5df4882 not found: ID does not exist"
Sep 16 18:47:49.311227 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.311159    2554 scope.go:117] "RemoveContainer" containerID="a08a45cf05fcf2ba623851978cce6330e8e5387cc93f5eb49a38f77926ae0313"
Sep 16 18:47:49.311379 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.311364    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a08a45cf05fcf2ba623851978cce6330e8e5387cc93f5eb49a38f77926ae0313"} err="failed to get container status \"a08a45cf05fcf2ba623851978cce6330e8e5387cc93f5eb49a38f77926ae0313\": rpc error: code = NotFound desc = could not find container \"a08a45cf05fcf2ba623851978cce6330e8e5387cc93f5eb49a38f77926ae0313\": container with ID starting with a08a45cf05fcf2ba623851978cce6330e8e5387cc93f5eb49a38f77926ae0313 not found: ID does not exist"
Sep 16 18:47:49.311421 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.311379    2554 scope.go:117] "RemoveContainer" containerID="8a225a3ccb6947cff5863f2df1931c915610801c11c1e88af802e6489bb183a1"
Sep 16 18:47:49.311591 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.311576    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a225a3ccb6947cff5863f2df1931c915610801c11c1e88af802e6489bb183a1"} err="failed to get container status \"8a225a3ccb6947cff5863f2df1931c915610801c11c1e88af802e6489bb183a1\": rpc error: code = NotFound desc = could not find container \"8a225a3ccb6947cff5863f2df1931c915610801c11c1e88af802e6489bb183a1\": container with ID starting with 8a225a3ccb6947cff5863f2df1931c915610801c11c1e88af802e6489bb183a1 not found: ID does not exist"
Sep 16 18:47:49.311632 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.311592    2554 scope.go:117] "RemoveContainer" containerID="98140a5d9415b0f70c10360dec70dce7e7aab74e4625bec8ba8510ec757e9801"
Sep 16 18:47:49.311833 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.311815    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98140a5d9415b0f70c10360dec70dce7e7aab74e4625bec8ba8510ec757e9801"} err="failed to get container status \"98140a5d9415b0f70c10360dec70dce7e7aab74e4625bec8ba8510ec757e9801\": rpc error: code = NotFound desc = could not find container \"98140a5d9415b0f70c10360dec70dce7e7aab74e4625bec8ba8510ec757e9801\": container with ID starting with 98140a5d9415b0f70c10360dec70dce7e7aab74e4625bec8ba8510ec757e9801 not found: ID does not exist"
Sep 16 18:47:49.393277 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.393251    2554 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-cluster-tls-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:49.393277 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.393273    2554 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-metrics-client-ca\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:49.393391 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.393284    2554 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-config-out\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:49.393391 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.393293    2554 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-alertmanager-main-db\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:49.393391 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.393302    2554 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:49.393391 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.393312    2554 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-config-volume\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:49.393391 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.393320    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrrrz\" (UniqueName: \"kubernetes.io/projected/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-kube-api-access-vrrrz\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:49.393391 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.393329    2554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:49.393391 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.393337    2554 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-web-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:49.393391 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.393345    2554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:49.393391 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.393354    2554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-secret-alertmanager-main-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:49.393391 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.393362    2554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:49.393391 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.393372    2554 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3adb4633-501b-4a6d-9a15-9ecbe0068e9f-tls-assets\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:49.581671 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.581493    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:47:49.587226 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.587199    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:47:49.632092 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.632058    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:47:49.632466 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.632448    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerName="alertmanager"
Sep 16 18:47:49.632550 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.632468    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerName="alertmanager"
Sep 16 18:47:49.632550 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.632492    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2688504b-6527-42f6-a904-8ef74b146bc7" containerName="registry"
Sep 16 18:47:49.632550 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.632500    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2688504b-6527-42f6-a904-8ef74b146bc7" containerName="registry"
Sep 16 18:47:49.632550 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.632509    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerName="kube-rbac-proxy-web"
Sep 16 18:47:49.632550 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.632517    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerName="kube-rbac-proxy-web"
Sep 16 18:47:49.632550 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.632537    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerName="prom-label-proxy"
Sep 16 18:47:49.632550 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.632545    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerName="prom-label-proxy"
Sep 16 18:47:49.632919 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.632556    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerName="config-reloader"
Sep 16 18:47:49.632919 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.632565    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerName="config-reloader"
Sep 16 18:47:49.632919 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.632574    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerName="kube-rbac-proxy"
Sep 16 18:47:49.632919 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.632583    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerName="kube-rbac-proxy"
Sep 16 18:47:49.632919 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.632593    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerName="init-config-reloader"
Sep 16 18:47:49.632919 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.632601    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerName="init-config-reloader"
Sep 16 18:47:49.632919 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.632612    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerName="kube-rbac-proxy-metric"
Sep 16 18:47:49.632919 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.632621    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerName="kube-rbac-proxy-metric"
Sep 16 18:47:49.632919 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.632693    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerName="alertmanager"
Sep 16 18:47:49.632919 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.632705    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerName="prom-label-proxy"
Sep 16 18:47:49.632919 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.632730    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerName="kube-rbac-proxy-web"
Sep 16 18:47:49.632919 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.632740    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerName="kube-rbac-proxy-metric"
Sep 16 18:47:49.632919 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.632751    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerName="kube-rbac-proxy"
Sep 16 18:47:49.632919 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.632760    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" containerName="config-reloader"
Sep 16 18:47:49.632919 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.632770    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2688504b-6527-42f6-a904-8ef74b146bc7" containerName="registry"
Sep 16 18:47:49.636402 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.636383    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.639606 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.639578    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 16 18:47:49.639606 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.639581    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 16 18:47:49.639944 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.639924    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 16 18:47:49.640029 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.639948    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 16 18:47:49.640029 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.639980    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 16 18:47:49.640029 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.639995    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 16 18:47:49.640029 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.640016    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jkm5h\""
Sep 16 18:47:49.640513 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.640499    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 16 18:47:49.642112 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.642091    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 16 18:47:49.647950 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.647931    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 16 18:47:49.664874 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.664849    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:47:49.695382 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.695353    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/70271f51-1ee3-4393-a70c-ede0c7fa4229-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.695560 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.695397    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/70271f51-1ee3-4393-a70c-ede0c7fa4229-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.695560 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.695421    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/70271f51-1ee3-4393-a70c-ede0c7fa4229-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.695560 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.695497    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/70271f51-1ee3-4393-a70c-ede0c7fa4229-config-volume\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.695560 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.695525    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70271f51-1ee3-4393-a70c-ede0c7fa4229-config-out\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.695560 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.695553    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70271f51-1ee3-4393-a70c-ede0c7fa4229-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.695749 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.695592    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70271f51-1ee3-4393-a70c-ede0c7fa4229-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.695749 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.695613    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70271f51-1ee3-4393-a70c-ede0c7fa4229-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.695749 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.695629    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9ngc\" (UniqueName: \"kubernetes.io/projected/70271f51-1ee3-4393-a70c-ede0c7fa4229-kube-api-access-v9ngc\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.695749 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.695657    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70271f51-1ee3-4393-a70c-ede0c7fa4229-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.695749 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.695690    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70271f51-1ee3-4393-a70c-ede0c7fa4229-tls-assets\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.695749 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.695737    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/70271f51-1ee3-4393-a70c-ede0c7fa4229-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.695978 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.695808    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70271f51-1ee3-4393-a70c-ede0c7fa4229-web-config\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.796420 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.796386    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70271f51-1ee3-4393-a70c-ede0c7fa4229-config-out\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.796420 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.796424    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70271f51-1ee3-4393-a70c-ede0c7fa4229-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.796595 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.796524    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70271f51-1ee3-4393-a70c-ede0c7fa4229-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.796595 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.796548    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70271f51-1ee3-4393-a70c-ede0c7fa4229-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.796595 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.796566    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9ngc\" (UniqueName: \"kubernetes.io/projected/70271f51-1ee3-4393-a70c-ede0c7fa4229-kube-api-access-v9ngc\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.796595 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.796587    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70271f51-1ee3-4393-a70c-ede0c7fa4229-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.796869 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.796608    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70271f51-1ee3-4393-a70c-ede0c7fa4229-tls-assets\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.796869 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.796640    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/70271f51-1ee3-4393-a70c-ede0c7fa4229-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.796869 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.796680    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70271f51-1ee3-4393-a70c-ede0c7fa4229-web-config\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.796869 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.796810    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/70271f51-1ee3-4393-a70c-ede0c7fa4229-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.796869 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.796862    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/70271f51-1ee3-4393-a70c-ede0c7fa4229-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.797132 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.796887    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/70271f51-1ee3-4393-a70c-ede0c7fa4229-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.797132 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.796920    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/70271f51-1ee3-4393-a70c-ede0c7fa4229-config-volume\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.797516 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.797489    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70271f51-1ee3-4393-a70c-ede0c7fa4229-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.797850 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.797798    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/70271f51-1ee3-4393-a70c-ede0c7fa4229-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.797969 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.797864    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70271f51-1ee3-4393-a70c-ede0c7fa4229-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.799816 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.799794    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/70271f51-1ee3-4393-a70c-ede0c7fa4229-config-volume\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.799971 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.799810    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/70271f51-1ee3-4393-a70c-ede0c7fa4229-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.800072 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.799956    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70271f51-1ee3-4393-a70c-ede0c7fa4229-tls-assets\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.800072 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.800028    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70271f51-1ee3-4393-a70c-ede0c7fa4229-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.800330 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.800309    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70271f51-1ee3-4393-a70c-ede0c7fa4229-web-config\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.800437 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.800420    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70271f51-1ee3-4393-a70c-ede0c7fa4229-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.800586 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.800569    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/70271f51-1ee3-4393-a70c-ede0c7fa4229-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.800640 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.800607    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/70271f51-1ee3-4393-a70c-ede0c7fa4229-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.801036 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.801017    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70271f51-1ee3-4393-a70c-ede0c7fa4229-config-out\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.805612 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.805595    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9ngc\" (UniqueName: \"kubernetes.io/projected/70271f51-1ee3-4393-a70c-ede0c7fa4229-kube-api-access-v9ngc\") pod \"alertmanager-main-0\" (UID: \"70271f51-1ee3-4393-a70c-ede0c7fa4229\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:49.946105 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:49.946002    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 16 18:47:50.232647 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:50.232536    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 16 18:47:50.235433 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:47:50.235409    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70271f51_1ee3_4393_a70c_ede0c7fa4229.slice/crio-d71330eb5c42a8361efcff4a058d1f5e35c104920a86635ce922a66d2635f7ec WatchSource:0}: Error finding container d71330eb5c42a8361efcff4a058d1f5e35c104920a86635ce922a66d2635f7ec: Status 404 returned error can't find the container with id d71330eb5c42a8361efcff4a058d1f5e35c104920a86635ce922a66d2635f7ec
Sep 16 18:47:50.256282 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:50.256258    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70271f51-1ee3-4393-a70c-ede0c7fa4229","Type":"ContainerStarted","Data":"d71330eb5c42a8361efcff4a058d1f5e35c104920a86635ce922a66d2635f7ec"}
Sep 16 18:47:50.442866 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:50.442833    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3adb4633-501b-4a6d-9a15-9ecbe0068e9f" path="/var/lib/kubelet/pods/3adb4633-501b-4a6d-9a15-9ecbe0068e9f/volumes"
Sep 16 18:47:51.261873 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:51.261844    2554 generic.go:356] "Generic (PLEG): container finished" podID="70271f51-1ee3-4393-a70c-ede0c7fa4229" containerID="bf375eedd95b9c77d7af812b96682fcbc0d8c735ba48b7c6b1dc147071a6de36" exitCode=0
Sep 16 18:47:51.262251 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:51.261915    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70271f51-1ee3-4393-a70c-ede0c7fa4229","Type":"ContainerDied","Data":"bf375eedd95b9c77d7af812b96682fcbc0d8c735ba48b7c6b1dc147071a6de36"}
Sep 16 18:47:51.905859 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:51.905781    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-65b8b8b754-fw6nc"]
Sep 16 18:47:51.909523 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:51.909500    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc"
Sep 16 18:47:51.914686 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:51.914659    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 16 18:47:51.914879 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:51.914859    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ace2021-c720-428d-8449-a06755fbc9a6-serving-certs-ca-bundle\") pod \"telemeter-client-65b8b8b754-fw6nc\" (UID: \"9ace2021-c720-428d-8449-a06755fbc9a6\") " pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc"
Sep 16 18:47:51.914919 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:51.914901    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9ace2021-c720-428d-8449-a06755fbc9a6-federate-client-tls\") pod \"telemeter-client-65b8b8b754-fw6nc\" (UID: \"9ace2021-c720-428d-8449-a06755fbc9a6\") " pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc"
Sep 16 18:47:51.914952 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:51.914935    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9ace2021-c720-428d-8449-a06755fbc9a6-telemeter-client-tls\") pod \"telemeter-client-65b8b8b754-fw6nc\" (UID: \"9ace2021-c720-428d-8449-a06755fbc9a6\") " pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc"
Sep 16 18:47:51.914997 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:51.914969    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbv4v\" (UniqueName: \"kubernetes.io/projected/9ace2021-c720-428d-8449-a06755fbc9a6-kube-api-access-fbv4v\") pod \"telemeter-client-65b8b8b754-fw6nc\" (UID: \"9ace2021-c720-428d-8449-a06755fbc9a6\") " pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc"
Sep 16 18:47:51.915063 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:51.915030    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ace2021-c720-428d-8449-a06755fbc9a6-metrics-client-ca\") pod \"telemeter-client-65b8b8b754-fw6nc\" (UID: \"9ace2021-c720-428d-8449-a06755fbc9a6\") " pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc"
Sep 16 18:47:51.915116 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:51.915077    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 16 18:47:51.915208 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:51.915087    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9ace2021-c720-428d-8449-a06755fbc9a6-secret-telemeter-client\") pod \"telemeter-client-65b8b8b754-fw6nc\" (UID: \"9ace2021-c720-428d-8449-a06755fbc9a6\") " pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc"
Sep 16 18:47:51.915304 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:51.915285    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 16 18:47:51.915384 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:51.915301    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 16 18:47:51.915384 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:51.915362    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-5px5d\""
Sep 16 18:47:51.915541 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:51.915512    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ace2021-c720-428d-8449-a06755fbc9a6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65b8b8b754-fw6nc\" (UID: \"9ace2021-c720-428d-8449-a06755fbc9a6\") " pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc"
Sep 16 18:47:51.915659 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:51.915591    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ace2021-c720-428d-8449-a06755fbc9a6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65b8b8b754-fw6nc\" (UID: \"9ace2021-c720-428d-8449-a06755fbc9a6\") " pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc"
Sep 16 18:47:51.915826 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:51.915809    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 16 18:47:51.924788 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:51.924736    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 16 18:47:51.935537 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:51.935509    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-65b8b8b754-fw6nc"]
Sep 16 18:47:52.016667 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.016645    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ace2021-c720-428d-8449-a06755fbc9a6-serving-certs-ca-bundle\") pod \"telemeter-client-65b8b8b754-fw6nc\" (UID: \"9ace2021-c720-428d-8449-a06755fbc9a6\") " pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc"
Sep 16 18:47:52.016800 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.016672    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9ace2021-c720-428d-8449-a06755fbc9a6-federate-client-tls\") pod \"telemeter-client-65b8b8b754-fw6nc\" (UID: \"9ace2021-c720-428d-8449-a06755fbc9a6\") " pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc"
Sep 16 18:47:52.016800 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.016692    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9ace2021-c720-428d-8449-a06755fbc9a6-telemeter-client-tls\") pod \"telemeter-client-65b8b8b754-fw6nc\" (UID: \"9ace2021-c720-428d-8449-a06755fbc9a6\") " pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc"
Sep 16 18:47:52.016800 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.016712    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbv4v\" (UniqueName: \"kubernetes.io/projected/9ace2021-c720-428d-8449-a06755fbc9a6-kube-api-access-fbv4v\") pod \"telemeter-client-65b8b8b754-fw6nc\" (UID: \"9ace2021-c720-428d-8449-a06755fbc9a6\") " pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc"
Sep 16 18:47:52.016800 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.016747    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ace2021-c720-428d-8449-a06755fbc9a6-metrics-client-ca\") pod \"telemeter-client-65b8b8b754-fw6nc\" (UID: \"9ace2021-c720-428d-8449-a06755fbc9a6\") " pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc"
Sep 16 18:47:52.016800 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.016766    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9ace2021-c720-428d-8449-a06755fbc9a6-secret-telemeter-client\") pod \"telemeter-client-65b8b8b754-fw6nc\" (UID: \"9ace2021-c720-428d-8449-a06755fbc9a6\") " pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc"
Sep 16 18:47:52.016800 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.016784    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ace2021-c720-428d-8449-a06755fbc9a6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65b8b8b754-fw6nc\" (UID: \"9ace2021-c720-428d-8449-a06755fbc9a6\") " pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc"
Sep 16 18:47:52.017123 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.016809    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ace2021-c720-428d-8449-a06755fbc9a6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65b8b8b754-fw6nc\" (UID: \"9ace2021-c720-428d-8449-a06755fbc9a6\") " pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc"
Sep 16 18:47:52.017498 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.017426    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ace2021-c720-428d-8449-a06755fbc9a6-serving-certs-ca-bundle\") pod \"telemeter-client-65b8b8b754-fw6nc\" (UID: \"9ace2021-c720-428d-8449-a06755fbc9a6\") " pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc"
Sep 16 18:47:52.017498 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.017460    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ace2021-c720-428d-8449-a06755fbc9a6-metrics-client-ca\") pod \"telemeter-client-65b8b8b754-fw6nc\" (UID: \"9ace2021-c720-428d-8449-a06755fbc9a6\") " pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc"
Sep 16 18:47:52.017644 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.017607    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ace2021-c720-428d-8449-a06755fbc9a6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65b8b8b754-fw6nc\" (UID: \"9ace2021-c720-428d-8449-a06755fbc9a6\") " pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc"
Sep 16 18:47:52.019193 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.019156    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9ace2021-c720-428d-8449-a06755fbc9a6-secret-telemeter-client\") pod \"telemeter-client-65b8b8b754-fw6nc\" (UID: \"9ace2021-c720-428d-8449-a06755fbc9a6\") " pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc"
Sep 16 18:47:52.019284 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.019268    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9ace2021-c720-428d-8449-a06755fbc9a6-federate-client-tls\") pod \"telemeter-client-65b8b8b754-fw6nc\" (UID: \"9ace2021-c720-428d-8449-a06755fbc9a6\") " pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc"
Sep 16 18:47:52.019389 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.019366    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ace2021-c720-428d-8449-a06755fbc9a6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65b8b8b754-fw6nc\" (UID: \"9ace2021-c720-428d-8449-a06755fbc9a6\") " pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc"
Sep 16 18:47:52.019426 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.019410    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9ace2021-c720-428d-8449-a06755fbc9a6-telemeter-client-tls\") pod \"telemeter-client-65b8b8b754-fw6nc\" (UID: \"9ace2021-c720-428d-8449-a06755fbc9a6\") " pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc"
Sep 16 18:47:52.030727 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.030699    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbv4v\" (UniqueName: \"kubernetes.io/projected/9ace2021-c720-428d-8449-a06755fbc9a6-kube-api-access-fbv4v\") pod \"telemeter-client-65b8b8b754-fw6nc\" (UID: \"9ace2021-c720-428d-8449-a06755fbc9a6\") " pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc"
Sep 16 18:47:52.227441 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.227389    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc"
Sep 16 18:47:52.236620 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.236597    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:47:52.237121 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.236911    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerName="prometheus" containerID="cri-o://7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64" gracePeriod=600
Sep 16 18:47:52.237121 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.236951    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerName="config-reloader" containerID="cri-o://1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68" gracePeriod=600
Sep 16 18:47:52.237121 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.236953    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerName="thanos-sidecar" containerID="cri-o://d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31" gracePeriod=600
Sep 16 18:47:52.237121 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.236953    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerName="kube-rbac-proxy-thanos" containerID="cri-o://9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127" gracePeriod=600
Sep 16 18:47:52.237121 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.236972    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerName="kube-rbac-proxy-web" containerID="cri-o://a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a" gracePeriod=600
Sep 16 18:47:52.237121 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.237065    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerName="kube-rbac-proxy" containerID="cri-o://4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010" gracePeriod=600
Sep 16 18:47:52.272967 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.272705    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70271f51-1ee3-4393-a70c-ede0c7fa4229","Type":"ContainerStarted","Data":"d8f6b1b811fd9b7dae43547921f881feed7edfc37e1dbbb633dd31119d6bd253"}
Sep 16 18:47:52.272967 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.272800    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70271f51-1ee3-4393-a70c-ede0c7fa4229","Type":"ContainerStarted","Data":"f3c5777a1d00dd12a0c7fcdc706890b2694afdbf6709b81dfd3a5f96239281dd"}
Sep 16 18:47:52.272967 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.272841    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70271f51-1ee3-4393-a70c-ede0c7fa4229","Type":"ContainerStarted","Data":"da1e26af4a8e5ddc8382187b7cbaf61d2b5ee008e0be8385c5893e2b19dbc599"}
Sep 16 18:47:52.272967 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.272879    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70271f51-1ee3-4393-a70c-ede0c7fa4229","Type":"ContainerStarted","Data":"1d7de1e9e577d96a6db70e2ac9756996903cc1d015c00f50e04158864bdc42d6"}
Sep 16 18:47:52.272967 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.272916    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70271f51-1ee3-4393-a70c-ede0c7fa4229","Type":"ContainerStarted","Data":"9866c73498734a28c660f2ba7bbe809f9412d711423f1e5d7b0ec4ec84a0bdee"}
Sep 16 18:47:52.272967 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.272952    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"70271f51-1ee3-4393-a70c-ede0c7fa4229","Type":"ContainerStarted","Data":"305f4f6dd5771db19ad55020e0ab773054deb3595b8e3eabe87af4eda6eba0bb"}
Sep 16 18:47:52.354533 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.354432    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.335971066 podStartE2EDuration="3.35441057s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.439504ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:52.352750501 +0000 UTC m=+182.428520134" watchObservedRunningTime="2026-09-16 18:47:52.35441057 +0000 UTC m=+182.430180203"
Sep 16 18:47:52.478754 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.478681    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:52.520912 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.520883    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-k8s-db\") pod \"c209e205-9e15-43c5-a592-98a5d0d7ad21\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") "
Sep 16 18:47:52.520912 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.520916    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-grpc-tls\") pod \"c209e205-9e15-43c5-a592-98a5d0d7ad21\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") "
Sep 16 18:47:52.521190 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.520940    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-prometheus-k8s-tls\") pod \"c209e205-9e15-43c5-a592-98a5d0d7ad21\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") "
Sep 16 18:47:52.521190 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.520964    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"c209e205-9e15-43c5-a592-98a5d0d7ad21\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") "
Sep 16 18:47:52.521190 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.520981    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-configmap-serving-certs-ca-bundle\") pod \"c209e205-9e15-43c5-a592-98a5d0d7ad21\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") "
Sep 16 18:47:52.521190 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.521050    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-k8s-rulefiles-2\") pod \"c209e205-9e15-43c5-a592-98a5d0d7ad21\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") "
Sep 16 18:47:52.521190 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.521085    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-thanos-prometheus-http-client-file\") pod \"c209e205-9e15-43c5-a592-98a5d0d7ad21\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") "
Sep 16 18:47:52.521190 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.521117    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-web-config\") pod \"c209e205-9e15-43c5-a592-98a5d0d7ad21\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") "
Sep 16 18:47:52.521190 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.521143    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-k8s-rulefiles-0\") pod \"c209e205-9e15-43c5-a592-98a5d0d7ad21\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") "
Sep 16 18:47:52.521190 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.521183    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-configmap-kubelet-serving-ca-bundle\") pod \"c209e205-9e15-43c5-a592-98a5d0d7ad21\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") "
Sep 16 18:47:52.521601 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.521213    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-trusted-ca-bundle\") pod \"c209e205-9e15-43c5-a592-98a5d0d7ad21\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") "
Sep 16 18:47:52.521601 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.521243    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-configmap-metrics-client-ca\") pod \"c209e205-9e15-43c5-a592-98a5d0d7ad21\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") "
Sep 16 18:47:52.521601 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.521279    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"c209e205-9e15-43c5-a592-98a5d0d7ad21\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") "
Sep 16 18:47:52.521601 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.521306    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qj9k4\" (UniqueName: \"kubernetes.io/projected/c209e205-9e15-43c5-a592-98a5d0d7ad21-kube-api-access-qj9k4\") pod \"c209e205-9e15-43c5-a592-98a5d0d7ad21\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") "
Sep 16 18:47:52.521601 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.521343    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "c209e205-9e15-43c5-a592-98a5d0d7ad21" (UID: "c209e205-9e15-43c5-a592-98a5d0d7ad21"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:52.521601 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.521350    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-metrics-client-certs\") pod \"c209e205-9e15-43c5-a592-98a5d0d7ad21\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") "
Sep 16 18:47:52.521601 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.521404    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-k8s-rulefiles-1\") pod \"c209e205-9e15-43c5-a592-98a5d0d7ad21\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") "
Sep 16 18:47:52.521601 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.521432    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c209e205-9e15-43c5-a592-98a5d0d7ad21-config-out\") pod \"c209e205-9e15-43c5-a592-98a5d0d7ad21\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") "
Sep 16 18:47:52.521601 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.521564    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c209e205-9e15-43c5-a592-98a5d0d7ad21-tls-assets\") pod \"c209e205-9e15-43c5-a592-98a5d0d7ad21\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") "
Sep 16 18:47:52.521601 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.521595    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-kube-rbac-proxy\") pod \"c209e205-9e15-43c5-a592-98a5d0d7ad21\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") "
Sep 16 18:47:52.523315 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.523133    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "c209e205-9e15-43c5-a592-98a5d0d7ad21" (UID: "c209e205-9e15-43c5-a592-98a5d0d7ad21"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:52.523428 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.523347    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "c209e205-9e15-43c5-a592-98a5d0d7ad21" (UID: "c209e205-9e15-43c5-a592-98a5d0d7ad21"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:52.523492 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.523414    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "c209e205-9e15-43c5-a592-98a5d0d7ad21" (UID: "c209e205-9e15-43c5-a592-98a5d0d7ad21"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:52.523556 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.523481    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "c209e205-9e15-43c5-a592-98a5d0d7ad21" (UID: "c209e205-9e15-43c5-a592-98a5d0d7ad21"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:52.524624 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.524086    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "c209e205-9e15-43c5-a592-98a5d0d7ad21" (UID: "c209e205-9e15-43c5-a592-98a5d0d7ad21"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:52.524784 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.524631    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "c209e205-9e15-43c5-a592-98a5d0d7ad21" (UID: "c209e205-9e15-43c5-a592-98a5d0d7ad21"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:52.524784 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.521625    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-config\") pod \"c209e205-9e15-43c5-a592-98a5d0d7ad21\" (UID: \"c209e205-9e15-43c5-a592-98a5d0d7ad21\") "
Sep 16 18:47:52.525376 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.525173    2554 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-grpc-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:52.525376 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.525196    2554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:52.525376 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.525211    2554 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:52.525376 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.525269    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:52.525376 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.525284    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:52.525376 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.525299    2554 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-metrics-client-certs\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:52.525376 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.525313    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:52.525376 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.525339    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "c209e205-9e15-43c5-a592-98a5d0d7ad21" (UID: "c209e205-9e15-43c5-a592-98a5d0d7ad21"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:52.525376 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.521620    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "c209e205-9e15-43c5-a592-98a5d0d7ad21" (UID: "c209e205-9e15-43c5-a592-98a5d0d7ad21"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:52.527952 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.525836    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "c209e205-9e15-43c5-a592-98a5d0d7ad21" (UID: "c209e205-9e15-43c5-a592-98a5d0d7ad21"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:52.527952 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.526858    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "c209e205-9e15-43c5-a592-98a5d0d7ad21" (UID: "c209e205-9e15-43c5-a592-98a5d0d7ad21"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:52.527952 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.526364    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "c209e205-9e15-43c5-a592-98a5d0d7ad21" (UID: "c209e205-9e15-43c5-a592-98a5d0d7ad21"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:47:52.531739 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.531694    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c209e205-9e15-43c5-a592-98a5d0d7ad21-config-out" (OuterVolumeSpecName: "config-out") pod "c209e205-9e15-43c5-a592-98a5d0d7ad21" (UID: "c209e205-9e15-43c5-a592-98a5d0d7ad21"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:47:52.531840 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.531737    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "c209e205-9e15-43c5-a592-98a5d0d7ad21" (UID: "c209e205-9e15-43c5-a592-98a5d0d7ad21"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:52.531840 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.531753    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-config" (OuterVolumeSpecName: "config") pod "c209e205-9e15-43c5-a592-98a5d0d7ad21" (UID: "c209e205-9e15-43c5-a592-98a5d0d7ad21"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:52.531928 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.531833    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "c209e205-9e15-43c5-a592-98a5d0d7ad21" (UID: "c209e205-9e15-43c5-a592-98a5d0d7ad21"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:52.531928 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.531904    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c209e205-9e15-43c5-a592-98a5d0d7ad21-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c209e205-9e15-43c5-a592-98a5d0d7ad21" (UID: "c209e205-9e15-43c5-a592-98a5d0d7ad21"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:52.532004 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.531974    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "c209e205-9e15-43c5-a592-98a5d0d7ad21" (UID: "c209e205-9e15-43c5-a592-98a5d0d7ad21"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:52.532490 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.532467    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c209e205-9e15-43c5-a592-98a5d0d7ad21-kube-api-access-qj9k4" (OuterVolumeSpecName: "kube-api-access-qj9k4") pod "c209e205-9e15-43c5-a592-98a5d0d7ad21" (UID: "c209e205-9e15-43c5-a592-98a5d0d7ad21"). InnerVolumeSpecName "kube-api-access-qj9k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:47:52.539856 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.539832    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-web-config" (OuterVolumeSpecName: "web-config") pod "c209e205-9e15-43c5-a592-98a5d0d7ad21" (UID: "c209e205-9e15-43c5-a592-98a5d0d7ad21"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:47:52.597434 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.597406    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-65b8b8b754-fw6nc"]
Sep 16 18:47:52.601578 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:47:52.601543    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ace2021_c720_428d_8449_a06755fbc9a6.slice/crio-d9cfc3869de4c67884bd45521c5220c40d12542e895c9e575c4e4fb62fcbbb8c WatchSource:0}: Error finding container d9cfc3869de4c67884bd45521c5220c40d12542e895c9e575c4e4fb62fcbbb8c: Status 404 returned error can't find the container with id d9cfc3869de4c67884bd45521c5220c40d12542e895c9e575c4e4fb62fcbbb8c
Sep 16 18:47:52.626053 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.626033    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-k8s-db\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:52.626136 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.626054    2554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:52.626136 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.626066    2554 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:52.626136 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.626075    2554 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-web-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:52.626136 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.626085    2554 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:52.626136 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.626094    2554 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:52.626136 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.626103    2554 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c209e205-9e15-43c5-a592-98a5d0d7ad21-configmap-metrics-client-ca\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:52.626136 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.626112    2554 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:52.626136 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.626120    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj9k4\" (UniqueName: \"kubernetes.io/projected/c209e205-9e15-43c5-a592-98a5d0d7ad21-kube-api-access-qj9k4\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:52.626136 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.626129    2554 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c209e205-9e15-43c5-a592-98a5d0d7ad21-config-out\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:52.626136 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.626137    2554 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c209e205-9e15-43c5-a592-98a5d0d7ad21-tls-assets\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:52.626439 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.626149    2554 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-secret-kube-rbac-proxy\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:52.626439 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:52.626157    2554 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/c209e205-9e15-43c5-a592-98a5d0d7ad21-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:47:53.279194 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.279146    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc" event={"ID":"9ace2021-c720-428d-8449-a06755fbc9a6","Type":"ContainerStarted","Data":"d9cfc3869de4c67884bd45521c5220c40d12542e895c9e575c4e4fb62fcbbb8c"}
Sep 16 18:47:53.282977 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.282950    2554 generic.go:356] "Generic (PLEG): container finished" podID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerID="9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127" exitCode=0
Sep 16 18:47:53.282977 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.282974    2554 generic.go:356] "Generic (PLEG): container finished" podID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerID="4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010" exitCode=0
Sep 16 18:47:53.282977 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.282982    2554 generic.go:356] "Generic (PLEG): container finished" podID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerID="a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a" exitCode=0
Sep 16 18:47:53.283238 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.282989    2554 generic.go:356] "Generic (PLEG): container finished" podID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerID="d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31" exitCode=0
Sep 16 18:47:53.283238 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.282994    2554 generic.go:356] "Generic (PLEG): container finished" podID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerID="1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68" exitCode=0
Sep 16 18:47:53.283238 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.283000    2554 generic.go:356] "Generic (PLEG): container finished" podID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerID="7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64" exitCode=0
Sep 16 18:47:53.283238 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.283057    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.283238 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.283077    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c209e205-9e15-43c5-a592-98a5d0d7ad21","Type":"ContainerDied","Data":"9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127"}
Sep 16 18:47:53.283238 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.283152    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c209e205-9e15-43c5-a592-98a5d0d7ad21","Type":"ContainerDied","Data":"4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010"}
Sep 16 18:47:53.283238 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.283202    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c209e205-9e15-43c5-a592-98a5d0d7ad21","Type":"ContainerDied","Data":"a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a"}
Sep 16 18:47:53.283238 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.283225    2554 scope.go:117] "RemoveContainer" containerID="9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127"
Sep 16 18:47:53.283617 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.283246    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c209e205-9e15-43c5-a592-98a5d0d7ad21","Type":"ContainerDied","Data":"d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31"}
Sep 16 18:47:53.283617 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.283287    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c209e205-9e15-43c5-a592-98a5d0d7ad21","Type":"ContainerDied","Data":"1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68"}
Sep 16 18:47:53.283617 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.283329    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c209e205-9e15-43c5-a592-98a5d0d7ad21","Type":"ContainerDied","Data":"7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64"}
Sep 16 18:47:53.283617 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.283369    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c209e205-9e15-43c5-a592-98a5d0d7ad21","Type":"ContainerDied","Data":"73501e2a60d7c0262b95fc9dac7413bbeb9cc1c7e542958797baee986dcbcf4f"}
Sep 16 18:47:53.293145 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.293044    2554 scope.go:117] "RemoveContainer" containerID="4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010"
Sep 16 18:47:53.302392 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.302324    2554 scope.go:117] "RemoveContainer" containerID="a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a"
Sep 16 18:47:53.310745 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.310706    2554 scope.go:117] "RemoveContainer" containerID="d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31"
Sep 16 18:47:53.317753 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.315825    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:47:53.322528 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.322510    2554 scope.go:117] "RemoveContainer" containerID="1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68"
Sep 16 18:47:53.330792 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.330774    2554 scope.go:117] "RemoveContainer" containerID="7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64"
Sep 16 18:47:53.335086 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.335065    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:47:53.339911 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.339883    2554 scope.go:117] "RemoveContainer" containerID="837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b"
Sep 16 18:47:53.349333 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.349319    2554 scope.go:117] "RemoveContainer" containerID="9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127"
Sep 16 18:47:53.349596 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:47:53.349574    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127\": container with ID starting with 9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127 not found: ID does not exist" containerID="9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127"
Sep 16 18:47:53.349676 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.349604    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127"} err="failed to get container status \"9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127\": rpc error: code = NotFound desc = could not find container \"9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127\": container with ID starting with 9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127 not found: ID does not exist"
Sep 16 18:47:53.349676 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.349623    2554 scope.go:117] "RemoveContainer" containerID="4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010"
Sep 16 18:47:53.349892 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:47:53.349874    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010\": container with ID starting with 4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010 not found: ID does not exist" containerID="4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010"
Sep 16 18:47:53.349935 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.349898    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010"} err="failed to get container status \"4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010\": rpc error: code = NotFound desc = could not find container \"4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010\": container with ID starting with 4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010 not found: ID does not exist"
Sep 16 18:47:53.349935 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.349914    2554 scope.go:117] "RemoveContainer" containerID="a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a"
Sep 16 18:47:53.350180 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:47:53.350166    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a\": container with ID starting with a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a not found: ID does not exist" containerID="a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a"
Sep 16 18:47:53.350224 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.350182    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a"} err="failed to get container status \"a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a\": rpc error: code = NotFound desc = could not find container \"a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a\": container with ID starting with a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a not found: ID does not exist"
Sep 16 18:47:53.350224 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.350194    2554 scope.go:117] "RemoveContainer" containerID="d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31"
Sep 16 18:47:53.350442 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:47:53.350425    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31\": container with ID starting with d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31 not found: ID does not exist" containerID="d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31"
Sep 16 18:47:53.350486 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.350445    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31"} err="failed to get container status \"d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31\": rpc error: code = NotFound desc = could not find container \"d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31\": container with ID starting with d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31 not found: ID does not exist"
Sep 16 18:47:53.350486 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.350457    2554 scope.go:117] "RemoveContainer" containerID="1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68"
Sep 16 18:47:53.350695 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:47:53.350678    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68\": container with ID starting with 1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68 not found: ID does not exist" containerID="1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68"
Sep 16 18:47:53.350756 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.350700    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68"} err="failed to get container status \"1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68\": rpc error: code = NotFound desc = could not find container \"1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68\": container with ID starting with 1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68 not found: ID does not exist"
Sep 16 18:47:53.350756 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.350734    2554 scope.go:117] "RemoveContainer" containerID="7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64"
Sep 16 18:47:53.350982 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:47:53.350967    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64\": container with ID starting with 7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64 not found: ID does not exist" containerID="7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64"
Sep 16 18:47:53.351030 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.350985    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64"} err="failed to get container status \"7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64\": rpc error: code = NotFound desc = could not find container \"7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64\": container with ID starting with 7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64 not found: ID does not exist"
Sep 16 18:47:53.351030 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.350998    2554 scope.go:117] "RemoveContainer" containerID="837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b"
Sep 16 18:47:53.351237 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:47:53.351221    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b\": container with ID starting with 837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b not found: ID does not exist" containerID="837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b"
Sep 16 18:47:53.351278 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.351241    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b"} err="failed to get container status \"837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b\": rpc error: code = NotFound desc = could not find container \"837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b\": container with ID starting with 837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b not found: ID does not exist"
Sep 16 18:47:53.351278 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.351254    2554 scope.go:117] "RemoveContainer" containerID="9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127"
Sep 16 18:47:53.351462 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.351448    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127"} err="failed to get container status \"9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127\": rpc error: code = NotFound desc = could not find container \"9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127\": container with ID starting with 9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127 not found: ID does not exist"
Sep 16 18:47:53.351502 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.351461    2554 scope.go:117] "RemoveContainer" containerID="4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010"
Sep 16 18:47:53.351661 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.351644    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010"} err="failed to get container status \"4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010\": rpc error: code = NotFound desc = could not find container \"4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010\": container with ID starting with 4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010 not found: ID does not exist"
Sep 16 18:47:53.351661 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.351660    2554 scope.go:117] "RemoveContainer" containerID="a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a"
Sep 16 18:47:53.351907 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.351889    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a"} err="failed to get container status \"a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a\": rpc error: code = NotFound desc = could not find container \"a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a\": container with ID starting with a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a not found: ID does not exist"
Sep 16 18:47:53.351955 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.351908    2554 scope.go:117] "RemoveContainer" containerID="d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31"
Sep 16 18:47:53.352118 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.352102    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31"} err="failed to get container status \"d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31\": rpc error: code = NotFound desc = could not find container \"d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31\": container with ID starting with d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31 not found: ID does not exist"
Sep 16 18:47:53.352118 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.352116    2554 scope.go:117] "RemoveContainer" containerID="1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68"
Sep 16 18:47:53.352346 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.352333    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68"} err="failed to get container status \"1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68\": rpc error: code = NotFound desc = could not find container \"1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68\": container with ID starting with 1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68 not found: ID does not exist"
Sep 16 18:47:53.352346 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.352345    2554 scope.go:117] "RemoveContainer" containerID="7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64"
Sep 16 18:47:53.352527 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.352512    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64"} err="failed to get container status \"7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64\": rpc error: code = NotFound desc = could not find container \"7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64\": container with ID starting with 7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64 not found: ID does not exist"
Sep 16 18:47:53.352527 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.352526    2554 scope.go:117] "RemoveContainer" containerID="837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b"
Sep 16 18:47:53.352738 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.352704    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b"} err="failed to get container status \"837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b\": rpc error: code = NotFound desc = could not find container \"837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b\": container with ID starting with 837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b not found: ID does not exist"
Sep 16 18:47:53.352788 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.352780    2554 scope.go:117] "RemoveContainer" containerID="9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127"
Sep 16 18:47:53.352999 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.352983    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127"} err="failed to get container status \"9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127\": rpc error: code = NotFound desc = could not find container \"9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127\": container with ID starting with 9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127 not found: ID does not exist"
Sep 16 18:47:53.352999 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.352998    2554 scope.go:117] "RemoveContainer" containerID="4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010"
Sep 16 18:47:53.353217 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.353198    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010"} err="failed to get container status \"4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010\": rpc error: code = NotFound desc = could not find container \"4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010\": container with ID starting with 4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010 not found: ID does not exist"
Sep 16 18:47:53.353262 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.353226    2554 scope.go:117] "RemoveContainer" containerID="a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a"
Sep 16 18:47:53.353427 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.353412    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a"} err="failed to get container status \"a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a\": rpc error: code = NotFound desc = could not find container \"a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a\": container with ID starting with a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a not found: ID does not exist"
Sep 16 18:47:53.353427 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.353425    2554 scope.go:117] "RemoveContainer" containerID="d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31"
Sep 16 18:47:53.353627 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.353611    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31"} err="failed to get container status \"d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31\": rpc error: code = NotFound desc = could not find container \"d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31\": container with ID starting with d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31 not found: ID does not exist"
Sep 16 18:47:53.353666 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.353628    2554 scope.go:117] "RemoveContainer" containerID="1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68"
Sep 16 18:47:53.353839 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.353821    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68"} err="failed to get container status \"1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68\": rpc error: code = NotFound desc = could not find container \"1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68\": container with ID starting with 1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68 not found: ID does not exist"
Sep 16 18:47:53.353839 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.353836    2554 scope.go:117] "RemoveContainer" containerID="7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64"
Sep 16 18:47:53.354044 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.354025    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64"} err="failed to get container status \"7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64\": rpc error: code = NotFound desc = could not find container \"7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64\": container with ID starting with 7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64 not found: ID does not exist"
Sep 16 18:47:53.354120 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.354046    2554 scope.go:117] "RemoveContainer" containerID="837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b"
Sep 16 18:47:53.354263 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.354249    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b"} err="failed to get container status \"837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b\": rpc error: code = NotFound desc = could not find container \"837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b\": container with ID starting with 837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b not found: ID does not exist"
Sep 16 18:47:53.354263 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.354262    2554 scope.go:117] "RemoveContainer" containerID="9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127"
Sep 16 18:47:53.354444 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.354430    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127"} err="failed to get container status \"9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127\": rpc error: code = NotFound desc = could not find container \"9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127\": container with ID starting with 9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127 not found: ID does not exist"
Sep 16 18:47:53.354444 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.354443    2554 scope.go:117] "RemoveContainer" containerID="4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010"
Sep 16 18:47:53.354618 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.354605    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010"} err="failed to get container status \"4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010\": rpc error: code = NotFound desc = could not find container \"4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010\": container with ID starting with 4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010 not found: ID does not exist"
Sep 16 18:47:53.354618 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.354617    2554 scope.go:117] "RemoveContainer" containerID="a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a"
Sep 16 18:47:53.354816 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.354802    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a"} err="failed to get container status \"a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a\": rpc error: code = NotFound desc = could not find container \"a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a\": container with ID starting with a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a not found: ID does not exist"
Sep 16 18:47:53.354856 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.354816    2554 scope.go:117] "RemoveContainer" containerID="d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31"
Sep 16 18:47:53.355015 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.354997    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31"} err="failed to get container status \"d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31\": rpc error: code = NotFound desc = could not find container \"d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31\": container with ID starting with d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31 not found: ID does not exist"
Sep 16 18:47:53.355057 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.355017    2554 scope.go:117] "RemoveContainer" containerID="1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68"
Sep 16 18:47:53.355244 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.355228    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68"} err="failed to get container status \"1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68\": rpc error: code = NotFound desc = could not find container \"1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68\": container with ID starting with 1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68 not found: ID does not exist"
Sep 16 18:47:53.355286 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.355243    2554 scope.go:117] "RemoveContainer" containerID="7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64"
Sep 16 18:47:53.355449 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.355431    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64"} err="failed to get container status \"7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64\": rpc error: code = NotFound desc = could not find container \"7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64\": container with ID starting with 7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64 not found: ID does not exist"
Sep 16 18:47:53.355507 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.355451    2554 scope.go:117] "RemoveContainer" containerID="837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b"
Sep 16 18:47:53.355678 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.355664    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b"} err="failed to get container status \"837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b\": rpc error: code = NotFound desc = could not find container \"837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b\": container with ID starting with 837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b not found: ID does not exist"
Sep 16 18:47:53.355740 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.355680    2554 scope.go:117] "RemoveContainer" containerID="9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127"
Sep 16 18:47:53.355897 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.355880    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127"} err="failed to get container status \"9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127\": rpc error: code = NotFound desc = could not find container \"9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127\": container with ID starting with 9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127 not found: ID does not exist"
Sep 16 18:47:53.355949 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.355899    2554 scope.go:117] "RemoveContainer" containerID="4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010"
Sep 16 18:47:53.356106 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.356090    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010"} err="failed to get container status \"4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010\": rpc error: code = NotFound desc = could not find container \"4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010\": container with ID starting with 4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010 not found: ID does not exist"
Sep 16 18:47:53.356146 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.356105    2554 scope.go:117] "RemoveContainer" containerID="a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a"
Sep 16 18:47:53.356329 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.356311    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a"} err="failed to get container status \"a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a\": rpc error: code = NotFound desc = could not find container \"a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a\": container with ID starting with a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a not found: ID does not exist"
Sep 16 18:47:53.356403 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.356332    2554 scope.go:117] "RemoveContainer" containerID="d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31"
Sep 16 18:47:53.356558 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.356542    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31"} err="failed to get container status \"d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31\": rpc error: code = NotFound desc = could not find container \"d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31\": container with ID starting with d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31 not found: ID does not exist"
Sep 16 18:47:53.356600 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.356558    2554 scope.go:117] "RemoveContainer" containerID="1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68"
Sep 16 18:47:53.356785 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.356767    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68"} err="failed to get container status \"1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68\": rpc error: code = NotFound desc = could not find container \"1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68\": container with ID starting with 1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68 not found: ID does not exist"
Sep 16 18:47:53.356837 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.356787    2554 scope.go:117] "RemoveContainer" containerID="7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64"
Sep 16 18:47:53.357015 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.356997    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64"} err="failed to get container status \"7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64\": rpc error: code = NotFound desc = could not find container \"7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64\": container with ID starting with 7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64 not found: ID does not exist"
Sep 16 18:47:53.357060 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.357017    2554 scope.go:117] "RemoveContainer" containerID="837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b"
Sep 16 18:47:53.357247 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.357228    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b"} err="failed to get container status \"837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b\": rpc error: code = NotFound desc = could not find container \"837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b\": container with ID starting with 837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b not found: ID does not exist"
Sep 16 18:47:53.357312 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.357249    2554 scope.go:117] "RemoveContainer" containerID="9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127"
Sep 16 18:47:53.357465 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.357449    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127"} err="failed to get container status \"9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127\": rpc error: code = NotFound desc = could not find container \"9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127\": container with ID starting with 9ae624131070c45a7f51530f960059ba3920e18b495b3754ba9c1b6e37ce3127 not found: ID does not exist"
Sep 16 18:47:53.357510 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.357464    2554 scope.go:117] "RemoveContainer" containerID="4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010"
Sep 16 18:47:53.357677 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.357660    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010"} err="failed to get container status \"4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010\": rpc error: code = NotFound desc = could not find container \"4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010\": container with ID starting with 4ece147ebc2587163ac0f834c6a0c78824a38937fe3e75086e86b4894dd53010 not found: ID does not exist"
Sep 16 18:47:53.357752 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.357678    2554 scope.go:117] "RemoveContainer" containerID="a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a"
Sep 16 18:47:53.357903 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.357885    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a"} err="failed to get container status \"a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a\": rpc error: code = NotFound desc = could not find container \"a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a\": container with ID starting with a23efb8b7ef383ec93df03514fa9990b2e74133e30fa5ec771b31a1413022c4a not found: ID does not exist"
Sep 16 18:47:53.357949 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.357903    2554 scope.go:117] "RemoveContainer" containerID="d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31"
Sep 16 18:47:53.358110 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.358091    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31"} err="failed to get container status \"d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31\": rpc error: code = NotFound desc = could not find container \"d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31\": container with ID starting with d474d4bce1d439899560fc7879aa329a8a8bfc1f5bdb967dbfadca7722a31a31 not found: ID does not exist"
Sep 16 18:47:53.358155 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.358110    2554 scope.go:117] "RemoveContainer" containerID="1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68"
Sep 16 18:47:53.358325 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.358311    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68"} err="failed to get container status \"1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68\": rpc error: code = NotFound desc = could not find container \"1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68\": container with ID starting with 1d10ca487d4810692022fb2c730af96b43a32729e596d60327abd33959aa3d68 not found: ID does not exist"
Sep 16 18:47:53.358365 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.358325    2554 scope.go:117] "RemoveContainer" containerID="7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64"
Sep 16 18:47:53.358533 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.358514    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64"} err="failed to get container status \"7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64\": rpc error: code = NotFound desc = could not find container \"7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64\": container with ID starting with 7efe8af922e9ddc22e1735d6b78e0b0e8a998da13675fa280e83c651ebbfef64 not found: ID does not exist"
Sep 16 18:47:53.358602 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.358535    2554 scope.go:117] "RemoveContainer" containerID="837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b"
Sep 16 18:47:53.358772 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.358756    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b"} err="failed to get container status \"837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b\": rpc error: code = NotFound desc = could not find container \"837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b\": container with ID starting with 837fb8c4f8aaca6746d21d68887d4754806c491d71fd19566dd6e7d76941093b not found: ID does not exist"
Sep 16 18:47:53.389405 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.389383    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:47:53.389790 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.389773    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerName="kube-rbac-proxy-thanos"
Sep 16 18:47:53.389889 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.389793    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerName="kube-rbac-proxy-thanos"
Sep 16 18:47:53.389889 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.389822    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerName="init-config-reloader"
Sep 16 18:47:53.389889 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.389831    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerName="init-config-reloader"
Sep 16 18:47:53.389889 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.389846    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerName="kube-rbac-proxy-web"
Sep 16 18:47:53.389889 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.389855    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerName="kube-rbac-proxy-web"
Sep 16 18:47:53.389889 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.389868    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerName="thanos-sidecar"
Sep 16 18:47:53.389889 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.389876    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerName="thanos-sidecar"
Sep 16 18:47:53.390224 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.389891    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerName="config-reloader"
Sep 16 18:47:53.390224 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.389900    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerName="config-reloader"
Sep 16 18:47:53.390224 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.389911    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerName="prometheus"
Sep 16 18:47:53.390224 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.389919    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerName="prometheus"
Sep 16 18:47:53.390224 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.389928    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerName="kube-rbac-proxy"
Sep 16 18:47:53.390224 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.389936    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerName="kube-rbac-proxy"
Sep 16 18:47:53.390224 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.390022    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerName="prometheus"
Sep 16 18:47:53.390224 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.390039    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerName="config-reloader"
Sep 16 18:47:53.390224 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.390048    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerName="thanos-sidecar"
Sep 16 18:47:53.390224 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.390057    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerName="kube-rbac-proxy"
Sep 16 18:47:53.390224 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.390066    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerName="kube-rbac-proxy-thanos"
Sep 16 18:47:53.390224 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.390076    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c209e205-9e15-43c5-a592-98a5d0d7ad21" containerName="kube-rbac-proxy-web"
Sep 16 18:47:53.395519 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.395502    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.410640 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.410621    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-8mq8tjp6d6n9u\""
Sep 16 18:47:53.410758 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.410669    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 16 18:47:53.410817 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.410795    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 16 18:47:53.410946 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.410927    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 16 18:47:53.411047 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.411028    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 16 18:47:53.411145 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.411055    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 16 18:47:53.411145 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.411125    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 16 18:47:53.411471 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.411452    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 16 18:47:53.413105 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.413081    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 16 18:47:53.413188 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.413121    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 16 18:47:53.413482 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.413467    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 16 18:47:53.414403 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.414385    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 16 18:47:53.415889 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.415870    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 16 18:47:53.415889 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.415884    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 16 18:47:53.416962 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.416891    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-fhlkj\""
Sep 16 18:47:53.422691 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.422671    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 16 18:47:53.434093 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.434075    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5a316a8e-4e14-4478-a5f1-58974dcc57d5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.434190 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.434107    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvd78\" (UniqueName: \"kubernetes.io/projected/5a316a8e-4e14-4478-a5f1-58974dcc57d5-kube-api-access-gvd78\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.434190 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.434136    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a316a8e-4e14-4478-a5f1-58974dcc57d5-config-out\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.434190 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.434161    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5a316a8e-4e14-4478-a5f1-58974dcc57d5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.434324 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.434248    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a316a8e-4e14-4478-a5f1-58974dcc57d5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.434324 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.434292    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a316a8e-4e14-4478-a5f1-58974dcc57d5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.434413 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.434352    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5a316a8e-4e14-4478-a5f1-58974dcc57d5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.434413 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.434378    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5a316a8e-4e14-4478-a5f1-58974dcc57d5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.434513 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.434451    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a316a8e-4e14-4478-a5f1-58974dcc57d5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.434513 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.434482    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5a316a8e-4e14-4478-a5f1-58974dcc57d5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.434513 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.434506    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a316a8e-4e14-4478-a5f1-58974dcc57d5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.434643 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.434536    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5a316a8e-4e14-4478-a5f1-58974dcc57d5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.434643 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.434611    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5a316a8e-4e14-4478-a5f1-58974dcc57d5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.434729 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.434648    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a316a8e-4e14-4478-a5f1-58974dcc57d5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.434729 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.434699    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5a316a8e-4e14-4478-a5f1-58974dcc57d5-config\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.434794 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.434741    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a316a8e-4e14-4478-a5f1-58974dcc57d5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.434794 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.434766    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5a316a8e-4e14-4478-a5f1-58974dcc57d5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.434876 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.434796    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a316a8e-4e14-4478-a5f1-58974dcc57d5-web-config\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.434876 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.434860    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a316a8e-4e14-4478-a5f1-58974dcc57d5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.434950 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.434897    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5a316a8e-4e14-4478-a5f1-58974dcc57d5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.448865 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.448846    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:47:53.536402 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.536318    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a316a8e-4e14-4478-a5f1-58974dcc57d5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.536402 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.536374    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5a316a8e-4e14-4478-a5f1-58974dcc57d5-config\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.536628 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.536476    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a316a8e-4e14-4478-a5f1-58974dcc57d5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.536628 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.536505    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5a316a8e-4e14-4478-a5f1-58974dcc57d5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.536628 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.536537    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a316a8e-4e14-4478-a5f1-58974dcc57d5-web-config\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.536628 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.536575    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a316a8e-4e14-4478-a5f1-58974dcc57d5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.536628 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.536612    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5a316a8e-4e14-4478-a5f1-58974dcc57d5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.536916 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.536664    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5a316a8e-4e14-4478-a5f1-58974dcc57d5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.536916 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.536690    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvd78\" (UniqueName: \"kubernetes.io/projected/5a316a8e-4e14-4478-a5f1-58974dcc57d5-kube-api-access-gvd78\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.536916 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.536728    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a316a8e-4e14-4478-a5f1-58974dcc57d5-config-out\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.536916 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.536759    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5a316a8e-4e14-4478-a5f1-58974dcc57d5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.536916 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.536797    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a316a8e-4e14-4478-a5f1-58974dcc57d5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.536916 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.536834    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a316a8e-4e14-4478-a5f1-58974dcc57d5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.536916 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.536867    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5a316a8e-4e14-4478-a5f1-58974dcc57d5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.536916 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.536891    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5a316a8e-4e14-4478-a5f1-58974dcc57d5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.537283 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.536924    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a316a8e-4e14-4478-a5f1-58974dcc57d5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.537283 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.536951    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5a316a8e-4e14-4478-a5f1-58974dcc57d5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.537283 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.536974    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a316a8e-4e14-4478-a5f1-58974dcc57d5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.537283 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.537005    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5a316a8e-4e14-4478-a5f1-58974dcc57d5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.537283 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.537035    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5a316a8e-4e14-4478-a5f1-58974dcc57d5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.537283 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.537142    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a316a8e-4e14-4478-a5f1-58974dcc57d5-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.537283 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.537278    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a316a8e-4e14-4478-a5f1-58974dcc57d5-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.537626 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.537333    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5a316a8e-4e14-4478-a5f1-58974dcc57d5-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.537859 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.537832    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5a316a8e-4e14-4478-a5f1-58974dcc57d5-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.538280 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.538183    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a316a8e-4e14-4478-a5f1-58974dcc57d5-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.539542 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.539456    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5a316a8e-4e14-4478-a5f1-58974dcc57d5-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.540417 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.540122    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5a316a8e-4e14-4478-a5f1-58974dcc57d5-config-out\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.540417 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.540367    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5a316a8e-4e14-4478-a5f1-58974dcc57d5-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.540567 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.540420    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5a316a8e-4e14-4478-a5f1-58974dcc57d5-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.541760 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.541705    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5a316a8e-4e14-4478-a5f1-58974dcc57d5-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.542274 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.542250    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5a316a8e-4e14-4478-a5f1-58974dcc57d5-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.542387 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.542281    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5a316a8e-4e14-4478-a5f1-58974dcc57d5-web-config\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.542554 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.542532    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5a316a8e-4e14-4478-a5f1-58974dcc57d5-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.543166 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.542996    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5a316a8e-4e14-4478-a5f1-58974dcc57d5-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.543166 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.543056    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5a316a8e-4e14-4478-a5f1-58974dcc57d5-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.543166 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.543092    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5a316a8e-4e14-4478-a5f1-58974dcc57d5-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.543353 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.543196    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5a316a8e-4e14-4478-a5f1-58974dcc57d5-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.543353 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.543223    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5a316a8e-4e14-4478-a5f1-58974dcc57d5-config\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.544855 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.544834    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5a316a8e-4e14-4478-a5f1-58974dcc57d5-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.549075 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.549053    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvd78\" (UniqueName: \"kubernetes.io/projected/5a316a8e-4e14-4478-a5f1-58974dcc57d5-kube-api-access-gvd78\") pod \"prometheus-k8s-0\" (UID: \"5a316a8e-4e14-4478-a5f1-58974dcc57d5\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:53.706119 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:53.706070    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:47:54.015619 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:54.015406    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 16 18:47:54.229311 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:47:54.229277    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a316a8e_4e14_4478_a5f1_58974dcc57d5.slice/crio-35d4024987898b7e59987bd16db33854fcb2b18d844d9c5f37dc4bc45bc1cb28 WatchSource:0}: Error finding container 35d4024987898b7e59987bd16db33854fcb2b18d844d9c5f37dc4bc45bc1cb28: Status 404 returned error can't find the container with id 35d4024987898b7e59987bd16db33854fcb2b18d844d9c5f37dc4bc45bc1cb28
Sep 16 18:47:54.291471 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:54.291443    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a316a8e-4e14-4478-a5f1-58974dcc57d5","Type":"ContainerStarted","Data":"35d4024987898b7e59987bd16db33854fcb2b18d844d9c5f37dc4bc45bc1cb28"}
Sep 16 18:47:54.445203 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:54.445172    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c209e205-9e15-43c5-a592-98a5d0d7ad21" path="/var/lib/kubelet/pods/c209e205-9e15-43c5-a592-98a5d0d7ad21/volumes"
Sep 16 18:47:55.297683 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:55.297651    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc" event={"ID":"9ace2021-c720-428d-8449-a06755fbc9a6","Type":"ContainerStarted","Data":"9b086092642e9474d4fe5c9693e99372009a72dce1d0994ebd9667633430544d"}
Sep 16 18:47:55.298146 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:55.297708    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc" event={"ID":"9ace2021-c720-428d-8449-a06755fbc9a6","Type":"ContainerStarted","Data":"73221627b165a00ad18ad2a24ad825a56d49a027a62fc0b6b34826c11b898a20"}
Sep 16 18:47:55.298146 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:55.297747    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc" event={"ID":"9ace2021-c720-428d-8449-a06755fbc9a6","Type":"ContainerStarted","Data":"d3cbba285235e4ea9951165cb073b9ea177f2b5a3e1d0f523a6dca076d84dde0"}
Sep 16 18:47:55.299190 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:55.299162    2554 generic.go:356] "Generic (PLEG): container finished" podID="5a316a8e-4e14-4478-a5f1-58974dcc57d5" containerID="0f3c934b6a2a742cee9d8c89f7c02e34e930a7636cb1993172172ac3707929e0" exitCode=0
Sep 16 18:47:55.299246 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:55.299229    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a316a8e-4e14-4478-a5f1-58974dcc57d5","Type":"ContainerDied","Data":"0f3c934b6a2a742cee9d8c89f7c02e34e930a7636cb1993172172ac3707929e0"}
Sep 16 18:47:55.335087 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:55.335036    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-65b8b8b754-fw6nc" podStartSLOduration=2.657184403 podStartE2EDuration="4.33501788s" totalImagesPullingTime="1.677833477s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:55.333688914 +0000 UTC m=+185.409458587" watchObservedRunningTime="2026-09-16 18:47:55.33501788 +0000 UTC m=+185.410787511"
Sep 16 18:47:56.307223 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:56.307187    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a316a8e-4e14-4478-a5f1-58974dcc57d5","Type":"ContainerStarted","Data":"e40b5e915363a40bc36834ecfe11deaa833fe7b5008f2abd4c2e504a98411e6c"}
Sep 16 18:47:56.307660 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:56.307256    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a316a8e-4e14-4478-a5f1-58974dcc57d5","Type":"ContainerStarted","Data":"9962a01daa162e9d058cd515d452092eff110571bb68f97361101b78ae5f28ed"}
Sep 16 18:47:56.307660 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:56.307283    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a316a8e-4e14-4478-a5f1-58974dcc57d5","Type":"ContainerStarted","Data":"62ed4f05629973798dc23ec185b7a941f8a8ecfc9528a4f8ced333d9320b0b03"}
Sep 16 18:47:56.307660 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:56.307309    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a316a8e-4e14-4478-a5f1-58974dcc57d5","Type":"ContainerStarted","Data":"92fe7cff68b75369630057e6720777524844343c7a14176d81b8261bca0053e0"}
Sep 16 18:47:56.307660 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:56.307333    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a316a8e-4e14-4478-a5f1-58974dcc57d5","Type":"ContainerStarted","Data":"297790f06a49906261b7122641d039c362ac4aa70b43b0d203ee9a4c090d40c5"}
Sep 16 18:47:56.307660 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:56.307355    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5a316a8e-4e14-4478-a5f1-58974dcc57d5","Type":"ContainerStarted","Data":"787f5ab8ba97c0e193fd7fbec47e3a9f7fa55c03c78fb831d399fd4674fa432a"}
Sep 16 18:47:56.399298 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:56.399249    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.371667216 podStartE2EDuration="3.399233753s" totalImagesPullingTime="0s" totalInitContainerRuntime="27.566537ms" isStatefulPod=false podCreationTimestamp="2026-09-16 18:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:47:56.395242327 +0000 UTC m=+186.471011959" watchObservedRunningTime="2026-09-16 18:47:56.399233753 +0000 UTC m=+186.475003384"
Sep 16 18:47:58.706429 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:47:58.706378    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:53.706554 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:48:53.706527    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:53.721121 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:48:53.721096    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:48:54.560440 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:48:54.560412    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 16 18:49:50.300787 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:49:50.300752    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/2.log"
Sep 16 18:49:50.304458 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:49:50.304435    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/2.log"
Sep 16 18:49:50.307251 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:49:50.307232    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovn-acl-logging/0.log"
Sep 16 18:49:50.310940 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:49:50.310918    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovn-acl-logging/0.log"
Sep 16 18:49:50.311455 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:49:50.311437    2554 kubelet.go:1648] "Image garbage collection succeeded"
Sep 16 18:53:40.150669 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:53:40.150642    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-hx7p6"]
Sep 16 18:53:40.153672 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:53:40.153656    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-hx7p6"
Sep 16 18:53:40.156918 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:53:40.156896    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 18:53:40.160602 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:53:40.160586    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:53:40.160678 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:53:40.160635    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:53:40.161144 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:53:40.161123    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-vwxwg\""
Sep 16 18:53:40.170091 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:53:40.170071    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:53:40.217589 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:53:40.217569    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-hx7p6"]
Sep 16 18:53:40.291053 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:53:40.291033    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0c15d01-2444-4234-b12c-3e032012a314-model-cache\") pod \"s3-init-hx7p6\" (UID: \"c0c15d01-2444-4234-b12c-3e032012a314\") " pod="kserve/s3-init-hx7p6"
Sep 16 18:53:40.291147 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:53:40.291069    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqd9x\" (UniqueName: \"kubernetes.io/projected/c0c15d01-2444-4234-b12c-3e032012a314-kube-api-access-mqd9x\") pod \"s3-init-hx7p6\" (UID: \"c0c15d01-2444-4234-b12c-3e032012a314\") " pod="kserve/s3-init-hx7p6"
Sep 16 18:53:40.392304 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:53:40.392279    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0c15d01-2444-4234-b12c-3e032012a314-model-cache\") pod \"s3-init-hx7p6\" (UID: \"c0c15d01-2444-4234-b12c-3e032012a314\") " pod="kserve/s3-init-hx7p6"
Sep 16 18:53:40.392398 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:53:40.392317    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqd9x\" (UniqueName: \"kubernetes.io/projected/c0c15d01-2444-4234-b12c-3e032012a314-kube-api-access-mqd9x\") pod \"s3-init-hx7p6\" (UID: \"c0c15d01-2444-4234-b12c-3e032012a314\") " pod="kserve/s3-init-hx7p6"
Sep 16 18:53:40.392706 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:53:40.392688    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0c15d01-2444-4234-b12c-3e032012a314-model-cache\") pod \"s3-init-hx7p6\" (UID: \"c0c15d01-2444-4234-b12c-3e032012a314\") " pod="kserve/s3-init-hx7p6"
Sep 16 18:53:40.401731 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:53:40.401675    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqd9x\" (UniqueName: \"kubernetes.io/projected/c0c15d01-2444-4234-b12c-3e032012a314-kube-api-access-mqd9x\") pod \"s3-init-hx7p6\" (UID: \"c0c15d01-2444-4234-b12c-3e032012a314\") " pod="kserve/s3-init-hx7p6"
Sep 16 18:53:40.470909 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:53:40.470886    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-hx7p6"
Sep 16 18:53:40.748106 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:53:40.748084    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-hx7p6"]
Sep 16 18:53:40.751598 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:53:40.751572    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0c15d01_2444_4234_b12c_3e032012a314.slice/crio-226dd4a4bdc6e1e46a7e2d62792c152ad2e077b4642815e7b4888e7ec00ffd00 WatchSource:0}: Error finding container 226dd4a4bdc6e1e46a7e2d62792c152ad2e077b4642815e7b4888e7ec00ffd00: Status 404 returned error can't find the container with id 226dd4a4bdc6e1e46a7e2d62792c152ad2e077b4642815e7b4888e7ec00ffd00
Sep 16 18:53:40.753509 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:53:40.753490    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 18:53:41.664620 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:53:41.664582    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hx7p6" event={"ID":"c0c15d01-2444-4234-b12c-3e032012a314","Type":"ContainerStarted","Data":"226dd4a4bdc6e1e46a7e2d62792c152ad2e077b4642815e7b4888e7ec00ffd00"}
Sep 16 18:53:44.679310 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:53:44.679280    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hx7p6" event={"ID":"c0c15d01-2444-4234-b12c-3e032012a314","Type":"ContainerStarted","Data":"d8ae008b556ca9678cdeea8a03a3958faf73954c248a81ce0fbf92a12b90375e"}
Sep 16 18:53:46.689798 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:53:46.689766    2554 generic.go:356] "Generic (PLEG): container finished" podID="c0c15d01-2444-4234-b12c-3e032012a314" containerID="d8ae008b556ca9678cdeea8a03a3958faf73954c248a81ce0fbf92a12b90375e" exitCode=0
Sep 16 18:53:46.690246 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:53:46.689837    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hx7p6" event={"ID":"c0c15d01-2444-4234-b12c-3e032012a314","Type":"ContainerDied","Data":"d8ae008b556ca9678cdeea8a03a3958faf73954c248a81ce0fbf92a12b90375e"}
Sep 16 18:53:51.058031 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:53:51.058006    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Sep 16 18:53:51.721125 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:53:51.721090    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hx7p6" event={"ID":"c0c15d01-2444-4234-b12c-3e032012a314","Type":"ContainerStarted","Data":"a7f6da9a7fd5c758242ff3617dc321ff1d3f749a74f968ff09539e834ac146a4"}
Sep 16 18:53:51.737224 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:53:51.737139    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-hx7p6" podStartSLOduration=2.230403564 podStartE2EDuration="11.737120411s" totalImagesPullingTime="8.070289289s" totalInitContainerRuntime="1.436427558s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:53:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:53:51.737083992 +0000 UTC m=+541.812853625" watchObservedRunningTime="2026-09-16 18:53:51.737120411 +0000 UTC m=+541.812890046"
Sep 16 18:54:50.331411 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:50.331381    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/2.log"
Sep 16 18:54:50.333565 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:50.333543    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/2.log"
Sep 16 18:54:50.337218 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:50.337198    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovn-acl-logging/0.log"
Sep 16 18:54:50.339482 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:50.339465    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovn-acl-logging/0.log"
Sep 16 18:54:55.988419 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:55.988380    2554 generic.go:356] "Generic (PLEG): container finished" podID="c0c15d01-2444-4234-b12c-3e032012a314" containerID="a7f6da9a7fd5c758242ff3617dc321ff1d3f749a74f968ff09539e834ac146a4" exitCode=0
Sep 16 18:54:55.988820 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:55.988455    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hx7p6" event={"ID":"c0c15d01-2444-4234-b12c-3e032012a314","Type":"ContainerDied","Data":"a7f6da9a7fd5c758242ff3617dc321ff1d3f749a74f968ff09539e834ac146a4"}
Sep 16 18:54:57.118490 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:57.118467    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-hx7p6"
Sep 16 18:54:57.204581 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:57.204546    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqd9x\" (UniqueName: \"kubernetes.io/projected/c0c15d01-2444-4234-b12c-3e032012a314-kube-api-access-mqd9x\") pod \"c0c15d01-2444-4234-b12c-3e032012a314\" (UID: \"c0c15d01-2444-4234-b12c-3e032012a314\") "
Sep 16 18:54:57.204803 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:57.204605    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0c15d01-2444-4234-b12c-3e032012a314-model-cache\") pod \"c0c15d01-2444-4234-b12c-3e032012a314\" (UID: \"c0c15d01-2444-4234-b12c-3e032012a314\") "
Sep 16 18:54:57.206671 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:57.206644    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c15d01-2444-4234-b12c-3e032012a314-kube-api-access-mqd9x" (OuterVolumeSpecName: "kube-api-access-mqd9x") pod "c0c15d01-2444-4234-b12c-3e032012a314" (UID: "c0c15d01-2444-4234-b12c-3e032012a314"). InnerVolumeSpecName "kube-api-access-mqd9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:54:57.223281 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:57.223250    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0c15d01-2444-4234-b12c-3e032012a314-model-cache" (OuterVolumeSpecName: "model-cache") pod "c0c15d01-2444-4234-b12c-3e032012a314" (UID: "c0c15d01-2444-4234-b12c-3e032012a314"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:54:57.306026 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:57.305994    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqd9x\" (UniqueName: \"kubernetes.io/projected/c0c15d01-2444-4234-b12c-3e032012a314-kube-api-access-mqd9x\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:54:57.306026 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:57.306025    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0c15d01-2444-4234-b12c-3e032012a314-model-cache\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:54:57.998325 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:57.998296    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-hx7p6"
Sep 16 18:54:57.998490 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:57.998291    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-hx7p6" event={"ID":"c0c15d01-2444-4234-b12c-3e032012a314","Type":"ContainerDied","Data":"226dd4a4bdc6e1e46a7e2d62792c152ad2e077b4642815e7b4888e7ec00ffd00"}
Sep 16 18:54:57.998490 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:57.998411    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="226dd4a4bdc6e1e46a7e2d62792c152ad2e077b4642815e7b4888e7ec00ffd00"
Sep 16 18:54:58.730168 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:58.730141    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-cxd88"]
Sep 16 18:54:58.730538 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:58.730479    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0c15d01-2444-4234-b12c-3e032012a314" containerName="download-hf-model"
Sep 16 18:54:58.730538 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:58.730492    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0c15d01-2444-4234-b12c-3e032012a314" containerName="download-hf-model"
Sep 16 18:54:58.730538 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:58.730510    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0c15d01-2444-4234-b12c-3e032012a314" containerName="s3-init"
Sep 16 18:54:58.730538 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:58.730516    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0c15d01-2444-4234-b12c-3e032012a314" containerName="s3-init"
Sep 16 18:54:58.730665 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:58.730564    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0c15d01-2444-4234-b12c-3e032012a314" containerName="s3-init"
Sep 16 18:54:58.733569 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:58.733549    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cxd88"
Sep 16 18:54:58.735620 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:58.735568    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:54:58.735795 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:58.735776    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:54:58.735846 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:58.735799    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 16 18:54:58.736057 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:58.736041    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-vwxwg\""
Sep 16 18:54:58.741756 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:58.741735    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-cxd88"]
Sep 16 18:54:58.818853 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:58.818827    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dcsg\" (UniqueName: \"kubernetes.io/projected/7b79d4e8-021a-45d5-81ee-c35869e6fd08-kube-api-access-4dcsg\") pod \"seaweedfs-tls-custom-6644bf5d6c-cxd88\" (UID: \"7b79d4e8-021a-45d5-81ee-c35869e6fd08\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cxd88"
Sep 16 18:54:58.818958 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:58.818869    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7b79d4e8-021a-45d5-81ee-c35869e6fd08-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-cxd88\" (UID: \"7b79d4e8-021a-45d5-81ee-c35869e6fd08\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cxd88"
Sep 16 18:54:58.919694 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:58.919663    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dcsg\" (UniqueName: \"kubernetes.io/projected/7b79d4e8-021a-45d5-81ee-c35869e6fd08-kube-api-access-4dcsg\") pod \"seaweedfs-tls-custom-6644bf5d6c-cxd88\" (UID: \"7b79d4e8-021a-45d5-81ee-c35869e6fd08\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cxd88"
Sep 16 18:54:58.919832 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:58.919714    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7b79d4e8-021a-45d5-81ee-c35869e6fd08-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-cxd88\" (UID: \"7b79d4e8-021a-45d5-81ee-c35869e6fd08\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cxd88"
Sep 16 18:54:58.920081 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:58.920063    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7b79d4e8-021a-45d5-81ee-c35869e6fd08-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-cxd88\" (UID: \"7b79d4e8-021a-45d5-81ee-c35869e6fd08\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cxd88"
Sep 16 18:54:58.928571 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:58.928544    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dcsg\" (UniqueName: \"kubernetes.io/projected/7b79d4e8-021a-45d5-81ee-c35869e6fd08-kube-api-access-4dcsg\") pod \"seaweedfs-tls-custom-6644bf5d6c-cxd88\" (UID: \"7b79d4e8-021a-45d5-81ee-c35869e6fd08\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cxd88"
Sep 16 18:54:59.042902 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:59.042877    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cxd88"
Sep 16 18:54:59.311689 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:54:59.311664    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-cxd88"]
Sep 16 18:54:59.314334 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:54:59.314309    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b79d4e8_021a_45d5_81ee_c35869e6fd08.slice/crio-cbff2f981a66821cbc242d29b5e65c61c518eee782ffb8f9ba056b74a512833d WatchSource:0}: Error finding container cbff2f981a66821cbc242d29b5e65c61c518eee782ffb8f9ba056b74a512833d: Status 404 returned error can't find the container with id cbff2f981a66821cbc242d29b5e65c61c518eee782ffb8f9ba056b74a512833d
Sep 16 18:55:00.010462 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:00.010418    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cxd88" event={"ID":"7b79d4e8-021a-45d5-81ee-c35869e6fd08","Type":"ContainerStarted","Data":"cbff2f981a66821cbc242d29b5e65c61c518eee782ffb8f9ba056b74a512833d"}
Sep 16 18:55:03.026041 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:03.025999    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cxd88" event={"ID":"7b79d4e8-021a-45d5-81ee-c35869e6fd08","Type":"ContainerStarted","Data":"77d011df37f60ea51c6d38054ddb2ec84fe2a778e2350cd7aefd85261e084639"}
Sep 16 18:55:03.040956 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:03.040906    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cxd88" podStartSLOduration=2.259328449 podStartE2EDuration="5.04088896s" totalImagesPullingTime="2.781560511s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:03.039414544 +0000 UTC m=+613.115184175" watchObservedRunningTime="2026-09-16 18:55:03.04088896 +0000 UTC m=+613.116658597"
Sep 16 18:55:05.293371 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:05.293324    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-cxd88"]
Sep 16 18:55:05.293815 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:05.293500    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cxd88" podUID="7b79d4e8-021a-45d5-81ee-c35869e6fd08" containerName="seaweedfs-tls-custom" containerID="cri-o://77d011df37f60ea51c6d38054ddb2ec84fe2a778e2350cd7aefd85261e084639" gracePeriod=30
Sep 16 18:55:32.833399 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:32.833373    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cxd88"
Sep 16 18:55:32.892897 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:32.892823    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7b79d4e8-021a-45d5-81ee-c35869e6fd08-data\") pod \"7b79d4e8-021a-45d5-81ee-c35869e6fd08\" (UID: \"7b79d4e8-021a-45d5-81ee-c35869e6fd08\") "
Sep 16 18:55:32.892897 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:32.892874    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dcsg\" (UniqueName: \"kubernetes.io/projected/7b79d4e8-021a-45d5-81ee-c35869e6fd08-kube-api-access-4dcsg\") pod \"7b79d4e8-021a-45d5-81ee-c35869e6fd08\" (UID: \"7b79d4e8-021a-45d5-81ee-c35869e6fd08\") "
Sep 16 18:55:32.894146 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:32.894121    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b79d4e8-021a-45d5-81ee-c35869e6fd08-data" (OuterVolumeSpecName: "data") pod "7b79d4e8-021a-45d5-81ee-c35869e6fd08" (UID: "7b79d4e8-021a-45d5-81ee-c35869e6fd08"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:32.894910 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:32.894881    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b79d4e8-021a-45d5-81ee-c35869e6fd08-kube-api-access-4dcsg" (OuterVolumeSpecName: "kube-api-access-4dcsg") pod "7b79d4e8-021a-45d5-81ee-c35869e6fd08" (UID: "7b79d4e8-021a-45d5-81ee-c35869e6fd08"). InnerVolumeSpecName "kube-api-access-4dcsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:32.993817 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:32.993789    2554 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7b79d4e8-021a-45d5-81ee-c35869e6fd08-data\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:55:32.993817 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:32.993814    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dcsg\" (UniqueName: \"kubernetes.io/projected/7b79d4e8-021a-45d5-81ee-c35869e6fd08-kube-api-access-4dcsg\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:55:33.149625 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:33.149548    2554 generic.go:356] "Generic (PLEG): container finished" podID="7b79d4e8-021a-45d5-81ee-c35869e6fd08" containerID="77d011df37f60ea51c6d38054ddb2ec84fe2a778e2350cd7aefd85261e084639" exitCode=0
Sep 16 18:55:33.149625 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:33.149591    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cxd88"
Sep 16 18:55:33.149834 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:33.149628    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cxd88" event={"ID":"7b79d4e8-021a-45d5-81ee-c35869e6fd08","Type":"ContainerDied","Data":"77d011df37f60ea51c6d38054ddb2ec84fe2a778e2350cd7aefd85261e084639"}
Sep 16 18:55:33.149834 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:33.149681    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-cxd88" event={"ID":"7b79d4e8-021a-45d5-81ee-c35869e6fd08","Type":"ContainerDied","Data":"cbff2f981a66821cbc242d29b5e65c61c518eee782ffb8f9ba056b74a512833d"}
Sep 16 18:55:33.149834 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:33.149706    2554 scope.go:117] "RemoveContainer" containerID="77d011df37f60ea51c6d38054ddb2ec84fe2a778e2350cd7aefd85261e084639"
Sep 16 18:55:33.159421 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:33.159402    2554 scope.go:117] "RemoveContainer" containerID="77d011df37f60ea51c6d38054ddb2ec84fe2a778e2350cd7aefd85261e084639"
Sep 16 18:55:33.159670 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:55:33.159648    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77d011df37f60ea51c6d38054ddb2ec84fe2a778e2350cd7aefd85261e084639\": container with ID starting with 77d011df37f60ea51c6d38054ddb2ec84fe2a778e2350cd7aefd85261e084639 not found: ID does not exist" containerID="77d011df37f60ea51c6d38054ddb2ec84fe2a778e2350cd7aefd85261e084639"
Sep 16 18:55:33.159713 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:33.159676    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77d011df37f60ea51c6d38054ddb2ec84fe2a778e2350cd7aefd85261e084639"} err="failed to get container status \"77d011df37f60ea51c6d38054ddb2ec84fe2a778e2350cd7aefd85261e084639\": rpc error: code = NotFound desc = could not find container \"77d011df37f60ea51c6d38054ddb2ec84fe2a778e2350cd7aefd85261e084639\": container with ID starting with 77d011df37f60ea51c6d38054ddb2ec84fe2a778e2350cd7aefd85261e084639 not found: ID does not exist"
Sep 16 18:55:33.169140 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:33.169117    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-cxd88"]
Sep 16 18:55:33.172714 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:33.172694    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-cxd88"]
Sep 16 18:55:34.442922 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:34.442890    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b79d4e8-021a-45d5-81ee-c35869e6fd08" path="/var/lib/kubelet/pods/7b79d4e8-021a-45d5-81ee-c35869e6fd08/volumes"
Sep 16 18:55:34.971445 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:34.971409    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-2l78c"]
Sep 16 18:55:34.971834 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:34.971816    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b79d4e8-021a-45d5-81ee-c35869e6fd08" containerName="seaweedfs-tls-custom"
Sep 16 18:55:34.971921 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:34.971836    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b79d4e8-021a-45d5-81ee-c35869e6fd08" containerName="seaweedfs-tls-custom"
Sep 16 18:55:34.971984 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:34.971927    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b79d4e8-021a-45d5-81ee-c35869e6fd08" containerName="seaweedfs-tls-custom"
Sep 16 18:55:34.976313 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:34.976292    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-2l78c"
Sep 16 18:55:34.978430 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:34.978404    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 18:55:34.978545 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:34.978429    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 16 18:55:34.978545 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:34.978489    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:55:34.978829 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:34.978813    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-vwxwg\""
Sep 16 18:55:34.978880 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:34.978816    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:55:34.983172 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:34.983153    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-2l78c"]
Sep 16 18:55:35.006782 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:35.006712    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjqq4\" (UniqueName: \"kubernetes.io/projected/d3915d78-9c68-4e96-a820-6c7c0fd8a89f-kube-api-access-zjqq4\") pod \"s3-tls-init-custom-2l78c\" (UID: \"d3915d78-9c68-4e96-a820-6c7c0fd8a89f\") " pod="kserve/s3-tls-init-custom-2l78c"
Sep 16 18:55:35.006904 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:35.006802    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3915d78-9c68-4e96-a820-6c7c0fd8a89f-model-cache\") pod \"s3-tls-init-custom-2l78c\" (UID: \"d3915d78-9c68-4e96-a820-6c7c0fd8a89f\") " pod="kserve/s3-tls-init-custom-2l78c"
Sep 16 18:55:35.107599 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:35.107559    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3915d78-9c68-4e96-a820-6c7c0fd8a89f-model-cache\") pod \"s3-tls-init-custom-2l78c\" (UID: \"d3915d78-9c68-4e96-a820-6c7c0fd8a89f\") " pod="kserve/s3-tls-init-custom-2l78c"
Sep 16 18:55:35.107816 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:35.107672    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjqq4\" (UniqueName: \"kubernetes.io/projected/d3915d78-9c68-4e96-a820-6c7c0fd8a89f-kube-api-access-zjqq4\") pod \"s3-tls-init-custom-2l78c\" (UID: \"d3915d78-9c68-4e96-a820-6c7c0fd8a89f\") " pod="kserve/s3-tls-init-custom-2l78c"
Sep 16 18:55:35.107952 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:35.107931    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3915d78-9c68-4e96-a820-6c7c0fd8a89f-model-cache\") pod \"s3-tls-init-custom-2l78c\" (UID: \"d3915d78-9c68-4e96-a820-6c7c0fd8a89f\") " pod="kserve/s3-tls-init-custom-2l78c"
Sep 16 18:55:35.115099 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:35.115076    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjqq4\" (UniqueName: \"kubernetes.io/projected/d3915d78-9c68-4e96-a820-6c7c0fd8a89f-kube-api-access-zjqq4\") pod \"s3-tls-init-custom-2l78c\" (UID: \"d3915d78-9c68-4e96-a820-6c7c0fd8a89f\") " pod="kserve/s3-tls-init-custom-2l78c"
Sep 16 18:55:35.300260 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:35.300234    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-2l78c"
Sep 16 18:55:35.568475 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:35.568448    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-2l78c"]
Sep 16 18:55:35.571431 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:55:35.571396    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd3915d78_9c68_4e96_a820_6c7c0fd8a89f.slice/crio-29c8099ac0766a5c7264cdfe1aea884f660c01f7cd158cddc31b71867a7306f8 WatchSource:0}: Error finding container 29c8099ac0766a5c7264cdfe1aea884f660c01f7cd158cddc31b71867a7306f8: Status 404 returned error can't find the container with id 29c8099ac0766a5c7264cdfe1aea884f660c01f7cd158cddc31b71867a7306f8
Sep 16 18:55:36.164606 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:36.164569    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-2l78c" event={"ID":"d3915d78-9c68-4e96-a820-6c7c0fd8a89f","Type":"ContainerStarted","Data":"a05eead986e40f1e6ade402e1802664ba9fa37e693a5a488571958900e136165"}
Sep 16 18:55:36.164606 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:36.164614    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-2l78c" event={"ID":"d3915d78-9c68-4e96-a820-6c7c0fd8a89f","Type":"ContainerStarted","Data":"29c8099ac0766a5c7264cdfe1aea884f660c01f7cd158cddc31b71867a7306f8"}
Sep 16 18:55:37.170226 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:37.170191    2554 generic.go:356] "Generic (PLEG): container finished" podID="d3915d78-9c68-4e96-a820-6c7c0fd8a89f" containerID="a05eead986e40f1e6ade402e1802664ba9fa37e693a5a488571958900e136165" exitCode=0
Sep 16 18:55:37.170568 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:37.170280    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-2l78c" event={"ID":"d3915d78-9c68-4e96-a820-6c7c0fd8a89f","Type":"ContainerDied","Data":"a05eead986e40f1e6ade402e1802664ba9fa37e693a5a488571958900e136165"}
Sep 16 18:55:38.176730 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:38.176682    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-2l78c" event={"ID":"d3915d78-9c68-4e96-a820-6c7c0fd8a89f","Type":"ContainerStarted","Data":"d7b3e23ad79c97181360828bd14759659e4a64f23cd1da543090ba8401220e39"}
Sep 16 18:55:38.193320 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:38.193274    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-2l78c" podStartSLOduration=2.755270985 podStartE2EDuration="4.193258898s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.437987913s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:38.191987007 +0000 UTC m=+648.267756638" watchObservedRunningTime="2026-09-16 18:55:38.193258898 +0000 UTC m=+648.269028529"
Sep 16 18:55:44.205354 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:44.205316    2554 generic.go:356] "Generic (PLEG): container finished" podID="d3915d78-9c68-4e96-a820-6c7c0fd8a89f" containerID="d7b3e23ad79c97181360828bd14759659e4a64f23cd1da543090ba8401220e39" exitCode=0
Sep 16 18:55:44.205766 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:44.205364    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-2l78c" event={"ID":"d3915d78-9c68-4e96-a820-6c7c0fd8a89f","Type":"ContainerDied","Data":"d7b3e23ad79c97181360828bd14759659e4a64f23cd1da543090ba8401220e39"}
Sep 16 18:55:45.361876 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:45.361854    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-2l78c"
Sep 16 18:55:45.403482 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:45.403461    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3915d78-9c68-4e96-a820-6c7c0fd8a89f-model-cache\") pod \"d3915d78-9c68-4e96-a820-6c7c0fd8a89f\" (UID: \"d3915d78-9c68-4e96-a820-6c7c0fd8a89f\") "
Sep 16 18:55:45.403592 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:45.403504    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjqq4\" (UniqueName: \"kubernetes.io/projected/d3915d78-9c68-4e96-a820-6c7c0fd8a89f-kube-api-access-zjqq4\") pod \"d3915d78-9c68-4e96-a820-6c7c0fd8a89f\" (UID: \"d3915d78-9c68-4e96-a820-6c7c0fd8a89f\") "
Sep 16 18:55:45.405422 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:45.405399    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3915d78-9c68-4e96-a820-6c7c0fd8a89f-kube-api-access-zjqq4" (OuterVolumeSpecName: "kube-api-access-zjqq4") pod "d3915d78-9c68-4e96-a820-6c7c0fd8a89f" (UID: "d3915d78-9c68-4e96-a820-6c7c0fd8a89f"). InnerVolumeSpecName "kube-api-access-zjqq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:45.428397 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:45.428373    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3915d78-9c68-4e96-a820-6c7c0fd8a89f-model-cache" (OuterVolumeSpecName: "model-cache") pod "d3915d78-9c68-4e96-a820-6c7c0fd8a89f" (UID: "d3915d78-9c68-4e96-a820-6c7c0fd8a89f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:45.504864 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:45.504840    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3915d78-9c68-4e96-a820-6c7c0fd8a89f-model-cache\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:55:45.504962 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:45.504867    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjqq4\" (UniqueName: \"kubernetes.io/projected/d3915d78-9c68-4e96-a820-6c7c0fd8a89f-kube-api-access-zjqq4\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:55:46.215760 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:46.215704    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-2l78c" event={"ID":"d3915d78-9c68-4e96-a820-6c7c0fd8a89f","Type":"ContainerDied","Data":"29c8099ac0766a5c7264cdfe1aea884f660c01f7cd158cddc31b71867a7306f8"}
Sep 16 18:55:46.215760 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:46.215763    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-2l78c"
Sep 16 18:55:46.215944 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:46.215766    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29c8099ac0766a5c7264cdfe1aea884f660c01f7cd158cddc31b71867a7306f8"
Sep 16 18:55:46.798464 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:46.798427    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-knpk4"]
Sep 16 18:55:46.798847 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:46.798788    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3915d78-9c68-4e96-a820-6c7c0fd8a89f" containerName="s3-tls-init-custom"
Sep 16 18:55:46.798847 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:46.798799    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3915d78-9c68-4e96-a820-6c7c0fd8a89f" containerName="s3-tls-init-custom"
Sep 16 18:55:46.798847 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:46.798813    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3915d78-9c68-4e96-a820-6c7c0fd8a89f" containerName="download-hf-model"
Sep 16 18:55:46.798847 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:46.798818    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3915d78-9c68-4e96-a820-6c7c0fd8a89f" containerName="download-hf-model"
Sep 16 18:55:46.798971 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:46.798869    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3915d78-9c68-4e96-a820-6c7c0fd8a89f" containerName="s3-tls-init-custom"
Sep 16 18:55:46.801875 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:46.801856    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-knpk4"
Sep 16 18:55:46.803963 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:46.803940    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Sep 16 18:55:46.804080 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:46.803946    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 16 18:55:46.804080 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:46.803946    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 16 18:55:46.804196 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:46.804083    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 16 18:55:46.804196 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:46.804115    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-vwxwg\""
Sep 16 18:55:46.813307 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:46.813285    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-knpk4"]
Sep 16 18:55:46.917382 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:46.917342    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a82ff854-4e45-47c5-ba9d-26ce25c223c7-data\") pod \"seaweedfs-tls-serving-67b7477c6d-knpk4\" (UID: \"a82ff854-4e45-47c5-ba9d-26ce25c223c7\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-knpk4"
Sep 16 18:55:46.917577 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:46.917403    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/a82ff854-4e45-47c5-ba9d-26ce25c223c7-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-knpk4\" (UID: \"a82ff854-4e45-47c5-ba9d-26ce25c223c7\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-knpk4"
Sep 16 18:55:46.917577 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:46.917492    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzlnv\" (UniqueName: \"kubernetes.io/projected/a82ff854-4e45-47c5-ba9d-26ce25c223c7-kube-api-access-jzlnv\") pod \"seaweedfs-tls-serving-67b7477c6d-knpk4\" (UID: \"a82ff854-4e45-47c5-ba9d-26ce25c223c7\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-knpk4"
Sep 16 18:55:47.018455 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:47.018417    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/a82ff854-4e45-47c5-ba9d-26ce25c223c7-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-knpk4\" (UID: \"a82ff854-4e45-47c5-ba9d-26ce25c223c7\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-knpk4"
Sep 16 18:55:47.018455 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:47.018460    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzlnv\" (UniqueName: \"kubernetes.io/projected/a82ff854-4e45-47c5-ba9d-26ce25c223c7-kube-api-access-jzlnv\") pod \"seaweedfs-tls-serving-67b7477c6d-knpk4\" (UID: \"a82ff854-4e45-47c5-ba9d-26ce25c223c7\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-knpk4"
Sep 16 18:55:47.018655 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:47.018515    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a82ff854-4e45-47c5-ba9d-26ce25c223c7-data\") pod \"seaweedfs-tls-serving-67b7477c6d-knpk4\" (UID: \"a82ff854-4e45-47c5-ba9d-26ce25c223c7\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-knpk4"
Sep 16 18:55:47.018951 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:47.018921    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a82ff854-4e45-47c5-ba9d-26ce25c223c7-data\") pod \"seaweedfs-tls-serving-67b7477c6d-knpk4\" (UID: \"a82ff854-4e45-47c5-ba9d-26ce25c223c7\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-knpk4"
Sep 16 18:55:47.020963 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:47.020944    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/a82ff854-4e45-47c5-ba9d-26ce25c223c7-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-knpk4\" (UID: \"a82ff854-4e45-47c5-ba9d-26ce25c223c7\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-knpk4"
Sep 16 18:55:47.027583 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:47.027560    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzlnv\" (UniqueName: \"kubernetes.io/projected/a82ff854-4e45-47c5-ba9d-26ce25c223c7-kube-api-access-jzlnv\") pod \"seaweedfs-tls-serving-67b7477c6d-knpk4\" (UID: \"a82ff854-4e45-47c5-ba9d-26ce25c223c7\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-knpk4"
Sep 16 18:55:47.111147 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:47.111044    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-knpk4"
Sep 16 18:55:47.383751 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:47.383712    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-knpk4"]
Sep 16 18:55:47.386121 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:55:47.386094    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda82ff854_4e45_47c5_ba9d_26ce25c223c7.slice/crio-b6fbc51976b3c5e9cbab56a8e7ed1238dc2927cb7780b9be578b1a251c7b154a WatchSource:0}: Error finding container b6fbc51976b3c5e9cbab56a8e7ed1238dc2927cb7780b9be578b1a251c7b154a: Status 404 returned error can't find the container with id b6fbc51976b3c5e9cbab56a8e7ed1238dc2927cb7780b9be578b1a251c7b154a
Sep 16 18:55:48.225790 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:48.225712    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-knpk4" event={"ID":"a82ff854-4e45-47c5-ba9d-26ce25c223c7","Type":"ContainerStarted","Data":"7294028c1f53b1d726ee2aa0786b87a40f10d03151e27d6ccba48da335a22c23"}
Sep 16 18:55:48.226144 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:48.225809    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-knpk4" event={"ID":"a82ff854-4e45-47c5-ba9d-26ce25c223c7","Type":"ContainerStarted","Data":"b6fbc51976b3c5e9cbab56a8e7ed1238dc2927cb7780b9be578b1a251c7b154a"}
Sep 16 18:55:48.242957 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:48.242911    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-knpk4" podStartSLOduration=1.9889550539999998 podStartE2EDuration="2.242895046s" totalImagesPullingTime="253.939992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:48.242108396 +0000 UTC m=+658.317878018" watchObservedRunningTime="2026-09-16 18:55:48.242895046 +0000 UTC m=+658.318664678"
Sep 16 18:55:48.776556 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:48.776522    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-9dw49"]
Sep 16 18:55:48.781954 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:48.781932    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-9dw49"
Sep 16 18:55:48.783925 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:48.783901    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"hf-token\""
Sep 16 18:55:48.786818 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:48.786797    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-9dw49"]
Sep 16 18:55:48.834802 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:48.834767    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bxbp\" (UniqueName: \"kubernetes.io/projected/f7def41f-183c-4d2e-8138-1f5e26837461-kube-api-access-9bxbp\") pod \"s3-tls-init-serving-9dw49\" (UID: \"f7def41f-183c-4d2e-8138-1f5e26837461\") " pod="kserve/s3-tls-init-serving-9dw49"
Sep 16 18:55:48.835026 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:48.834816    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7def41f-183c-4d2e-8138-1f5e26837461-model-cache\") pod \"s3-tls-init-serving-9dw49\" (UID: \"f7def41f-183c-4d2e-8138-1f5e26837461\") " pod="kserve/s3-tls-init-serving-9dw49"
Sep 16 18:55:48.935630 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:48.935597    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7def41f-183c-4d2e-8138-1f5e26837461-model-cache\") pod \"s3-tls-init-serving-9dw49\" (UID: \"f7def41f-183c-4d2e-8138-1f5e26837461\") " pod="kserve/s3-tls-init-serving-9dw49"
Sep 16 18:55:48.935874 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:48.935676    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bxbp\" (UniqueName: \"kubernetes.io/projected/f7def41f-183c-4d2e-8138-1f5e26837461-kube-api-access-9bxbp\") pod \"s3-tls-init-serving-9dw49\" (UID: \"f7def41f-183c-4d2e-8138-1f5e26837461\") " pod="kserve/s3-tls-init-serving-9dw49"
Sep 16 18:55:48.936061 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:48.936037    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7def41f-183c-4d2e-8138-1f5e26837461-model-cache\") pod \"s3-tls-init-serving-9dw49\" (UID: \"f7def41f-183c-4d2e-8138-1f5e26837461\") " pod="kserve/s3-tls-init-serving-9dw49"
Sep 16 18:55:48.945798 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:48.945772    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bxbp\" (UniqueName: \"kubernetes.io/projected/f7def41f-183c-4d2e-8138-1f5e26837461-kube-api-access-9bxbp\") pod \"s3-tls-init-serving-9dw49\" (UID: \"f7def41f-183c-4d2e-8138-1f5e26837461\") " pod="kserve/s3-tls-init-serving-9dw49"
Sep 16 18:55:49.102358 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:49.102263    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-9dw49"
Sep 16 18:55:49.374251 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:49.374171    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-9dw49"]
Sep 16 18:55:49.377558 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:55:49.377527    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7def41f_183c_4d2e_8138_1f5e26837461.slice/crio-a373b08394f560c231d095bbd20e0e0a69e19e816d98f0eff678e048e5142e3d WatchSource:0}: Error finding container a373b08394f560c231d095bbd20e0e0a69e19e816d98f0eff678e048e5142e3d: Status 404 returned error can't find the container with id a373b08394f560c231d095bbd20e0e0a69e19e816d98f0eff678e048e5142e3d
Sep 16 18:55:50.237432 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:50.237386    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9dw49" event={"ID":"f7def41f-183c-4d2e-8138-1f5e26837461","Type":"ContainerStarted","Data":"0e1fdb59c750c9df222232b31559f7437c04552b033fe8f9f055f8d1b7e9d4f8"}
Sep 16 18:55:50.237626 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:50.237508    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9dw49" event={"ID":"f7def41f-183c-4d2e-8138-1f5e26837461","Type":"ContainerStarted","Data":"a373b08394f560c231d095bbd20e0e0a69e19e816d98f0eff678e048e5142e3d"}
Sep 16 18:55:51.244129 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:51.244041    2554 generic.go:356] "Generic (PLEG): container finished" podID="f7def41f-183c-4d2e-8138-1f5e26837461" containerID="0e1fdb59c750c9df222232b31559f7437c04552b033fe8f9f055f8d1b7e9d4f8" exitCode=0
Sep 16 18:55:51.244129 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:51.244099    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9dw49" event={"ID":"f7def41f-183c-4d2e-8138-1f5e26837461","Type":"ContainerDied","Data":"0e1fdb59c750c9df222232b31559f7437c04552b033fe8f9f055f8d1b7e9d4f8"}
Sep 16 18:55:52.250223 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:52.250192    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9dw49" event={"ID":"f7def41f-183c-4d2e-8138-1f5e26837461","Type":"ContainerStarted","Data":"530a4d0985d86f3ba37a7ddf50eaffe80f5ddbdf114ca785fa9725d9bd3ba23a"}
Sep 16 18:55:52.266439 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:52.266391    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-9dw49" podStartSLOduration=2.714301626 podStartE2EDuration="4.266374798s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.552073172s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:55:52.26540986 +0000 UTC m=+662.341179504" watchObservedRunningTime="2026-09-16 18:55:52.266374798 +0000 UTC m=+662.342144429"
Sep 16 18:55:57.276086 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:57.276046    2554 generic.go:356] "Generic (PLEG): container finished" podID="f7def41f-183c-4d2e-8138-1f5e26837461" containerID="530a4d0985d86f3ba37a7ddf50eaffe80f5ddbdf114ca785fa9725d9bd3ba23a" exitCode=0
Sep 16 18:55:57.276422 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:57.276111    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9dw49" event={"ID":"f7def41f-183c-4d2e-8138-1f5e26837461","Type":"ContainerDied","Data":"530a4d0985d86f3ba37a7ddf50eaffe80f5ddbdf114ca785fa9725d9bd3ba23a"}
Sep 16 18:55:58.415812 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:58.415788    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-9dw49"
Sep 16 18:55:58.519858 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:58.519833    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bxbp\" (UniqueName: \"kubernetes.io/projected/f7def41f-183c-4d2e-8138-1f5e26837461-kube-api-access-9bxbp\") pod \"f7def41f-183c-4d2e-8138-1f5e26837461\" (UID: \"f7def41f-183c-4d2e-8138-1f5e26837461\") "
Sep 16 18:55:58.520009 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:58.519903    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7def41f-183c-4d2e-8138-1f5e26837461-model-cache\") pod \"f7def41f-183c-4d2e-8138-1f5e26837461\" (UID: \"f7def41f-183c-4d2e-8138-1f5e26837461\") "
Sep 16 18:55:58.521896 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:58.521865    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7def41f-183c-4d2e-8138-1f5e26837461-kube-api-access-9bxbp" (OuterVolumeSpecName: "kube-api-access-9bxbp") pod "f7def41f-183c-4d2e-8138-1f5e26837461" (UID: "f7def41f-183c-4d2e-8138-1f5e26837461"). InnerVolumeSpecName "kube-api-access-9bxbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:55:58.541389 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:58.541332    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7def41f-183c-4d2e-8138-1f5e26837461-model-cache" (OuterVolumeSpecName: "model-cache") pod "f7def41f-183c-4d2e-8138-1f5e26837461" (UID: "f7def41f-183c-4d2e-8138-1f5e26837461"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:55:58.621428 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:58.621387    2554 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7def41f-183c-4d2e-8138-1f5e26837461-model-cache\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:55:58.621428 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:58.621415    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bxbp\" (UniqueName: \"kubernetes.io/projected/f7def41f-183c-4d2e-8138-1f5e26837461-kube-api-access-9bxbp\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:55:59.286000 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:59.285971    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-9dw49"
Sep 16 18:55:59.286243 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:59.286003    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-9dw49" event={"ID":"f7def41f-183c-4d2e-8138-1f5e26837461","Type":"ContainerDied","Data":"a373b08394f560c231d095bbd20e0e0a69e19e816d98f0eff678e048e5142e3d"}
Sep 16 18:55:59.286243 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:55:59.286034    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a373b08394f560c231d095bbd20e0e0a69e19e816d98f0eff678e048e5142e3d"
Sep 16 18:56:11.271250 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.271211    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"]
Sep 16 18:56:11.271659 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.271556    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7def41f-183c-4d2e-8138-1f5e26837461" containerName="s3-tls-init-serving"
Sep 16 18:56:11.271659 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.271568    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7def41f-183c-4d2e-8138-1f5e26837461" containerName="s3-tls-init-serving"
Sep 16 18:56:11.271659 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.271581    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7def41f-183c-4d2e-8138-1f5e26837461" containerName="download-hf-model"
Sep 16 18:56:11.271659 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.271586    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7def41f-183c-4d2e-8138-1f5e26837461" containerName="download-hf-model"
Sep 16 18:56:11.271659 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.271637    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7def41f-183c-4d2e-8138-1f5e26837461" containerName="s3-tls-init-serving"
Sep 16 18:56:11.275351 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.275329    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"
Sep 16 18:56:11.288966 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.288944    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Sep 16 18:56:11.289238 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.289213    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 18:56:11.289345 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.289245    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 18:56:11.289345 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.289292    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 18:56:11.289505 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.289488    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8gklb\""
Sep 16 18:56:11.289601 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.289504    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 18:56:11.289601 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.289517    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Sep 16 18:56:11.310085 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.310053    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"]
Sep 16 18:56:11.438241 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.438209    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63560122-fda1-4051-b4af-2b6bb701688d-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-74694d5594-2pqr7\" (UID: \"63560122-fda1-4051-b4af-2b6bb701688d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"
Sep 16 18:56:11.438419 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.438267    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63560122-fda1-4051-b4af-2b6bb701688d-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-74694d5594-2pqr7\" (UID: \"63560122-fda1-4051-b4af-2b6bb701688d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"
Sep 16 18:56:11.438419 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.438344    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63560122-fda1-4051-b4af-2b6bb701688d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-74694d5594-2pqr7\" (UID: \"63560122-fda1-4051-b4af-2b6bb701688d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"
Sep 16 18:56:11.438419 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.438371    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63560122-fda1-4051-b4af-2b6bb701688d-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-74694d5594-2pqr7\" (UID: \"63560122-fda1-4051-b4af-2b6bb701688d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"
Sep 16 18:56:11.438419 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.438395    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcps9\" (UniqueName: \"kubernetes.io/projected/63560122-fda1-4051-b4af-2b6bb701688d-kube-api-access-bcps9\") pod \"isvc-sklearn-batcher-predictor-74694d5594-2pqr7\" (UID: \"63560122-fda1-4051-b4af-2b6bb701688d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"
Sep 16 18:56:11.445966 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.445938    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj"]
Sep 16 18:56:11.449108 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.449092    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj"
Sep 16 18:56:11.451358 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.451336    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-579321-kube-rbac-proxy-sar-config\""
Sep 16 18:56:11.451539 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.451525    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-579321-predictor-serving-cert\""
Sep 16 18:56:11.451617 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.451550    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-579321\""
Sep 16 18:56:11.451617 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.451551    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-579321-dockercfg-czpkw\""
Sep 16 18:56:11.459709 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.459684    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj"]
Sep 16 18:56:11.540040 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.539951    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63560122-fda1-4051-b4af-2b6bb701688d-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-74694d5594-2pqr7\" (UID: \"63560122-fda1-4051-b4af-2b6bb701688d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"
Sep 16 18:56:11.540040 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.540010    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63560122-fda1-4051-b4af-2b6bb701688d-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-74694d5594-2pqr7\" (UID: \"63560122-fda1-4051-b4af-2b6bb701688d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"
Sep 16 18:56:11.540252 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.540045    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63560122-fda1-4051-b4af-2b6bb701688d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-74694d5594-2pqr7\" (UID: \"63560122-fda1-4051-b4af-2b6bb701688d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"
Sep 16 18:56:11.540252 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.540070    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63560122-fda1-4051-b4af-2b6bb701688d-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-74694d5594-2pqr7\" (UID: \"63560122-fda1-4051-b4af-2b6bb701688d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"
Sep 16 18:56:11.540252 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.540098    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcps9\" (UniqueName: \"kubernetes.io/projected/63560122-fda1-4051-b4af-2b6bb701688d-kube-api-access-bcps9\") pod \"isvc-sklearn-batcher-predictor-74694d5594-2pqr7\" (UID: \"63560122-fda1-4051-b4af-2b6bb701688d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"
Sep 16 18:56:11.540571 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.540548    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63560122-fda1-4051-b4af-2b6bb701688d-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-74694d5594-2pqr7\" (UID: \"63560122-fda1-4051-b4af-2b6bb701688d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"
Sep 16 18:56:11.540778 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.540744    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63560122-fda1-4051-b4af-2b6bb701688d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-74694d5594-2pqr7\" (UID: \"63560122-fda1-4051-b4af-2b6bb701688d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"
Sep 16 18:56:11.540837 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.540754    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63560122-fda1-4051-b4af-2b6bb701688d-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-74694d5594-2pqr7\" (UID: \"63560122-fda1-4051-b4af-2b6bb701688d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"
Sep 16 18:56:11.542545 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.542528    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63560122-fda1-4051-b4af-2b6bb701688d-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-74694d5594-2pqr7\" (UID: \"63560122-fda1-4051-b4af-2b6bb701688d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"
Sep 16 18:56:11.548157 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.548133    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcps9\" (UniqueName: \"kubernetes.io/projected/63560122-fda1-4051-b4af-2b6bb701688d-kube-api-access-bcps9\") pod \"isvc-sklearn-batcher-predictor-74694d5594-2pqr7\" (UID: \"63560122-fda1-4051-b4af-2b6bb701688d\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"
Sep 16 18:56:11.585743 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.585685    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"
Sep 16 18:56:11.641347 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.641314    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e03981-1083-4f6b-8004-317a2b6ed311-proxy-tls\") pod \"isvc-init-fail-579321-predictor-5d7f57b548-wpthj\" (UID: \"09e03981-1083-4f6b-8004-317a2b6ed311\") " pod="kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj"
Sep 16 18:56:11.641473 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.641402    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09e03981-1083-4f6b-8004-317a2b6ed311-kserve-provision-location\") pod \"isvc-init-fail-579321-predictor-5d7f57b548-wpthj\" (UID: \"09e03981-1083-4f6b-8004-317a2b6ed311\") " pod="kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj"
Sep 16 18:56:11.641473 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.641467    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e03981-1083-4f6b-8004-317a2b6ed311-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-579321-predictor-5d7f57b548-wpthj\" (UID: \"09e03981-1083-4f6b-8004-317a2b6ed311\") " pod="kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj"
Sep 16 18:56:11.641600 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.641560    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kv6k\" (UniqueName: \"kubernetes.io/projected/09e03981-1083-4f6b-8004-317a2b6ed311-kube-api-access-2kv6k\") pod \"isvc-init-fail-579321-predictor-5d7f57b548-wpthj\" (UID: \"09e03981-1083-4f6b-8004-317a2b6ed311\") " pod="kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj"
Sep 16 18:56:11.641654 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.641605    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09e03981-1083-4f6b-8004-317a2b6ed311-cabundle-cert\") pod \"isvc-init-fail-579321-predictor-5d7f57b548-wpthj\" (UID: \"09e03981-1083-4f6b-8004-317a2b6ed311\") " pod="kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj"
Sep 16 18:56:11.742277 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.742243    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09e03981-1083-4f6b-8004-317a2b6ed311-cabundle-cert\") pod \"isvc-init-fail-579321-predictor-5d7f57b548-wpthj\" (UID: \"09e03981-1083-4f6b-8004-317a2b6ed311\") " pod="kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj"
Sep 16 18:56:11.742508 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.742290    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e03981-1083-4f6b-8004-317a2b6ed311-proxy-tls\") pod \"isvc-init-fail-579321-predictor-5d7f57b548-wpthj\" (UID: \"09e03981-1083-4f6b-8004-317a2b6ed311\") " pod="kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj"
Sep 16 18:56:11.742508 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.742328    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09e03981-1083-4f6b-8004-317a2b6ed311-kserve-provision-location\") pod \"isvc-init-fail-579321-predictor-5d7f57b548-wpthj\" (UID: \"09e03981-1083-4f6b-8004-317a2b6ed311\") " pod="kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj"
Sep 16 18:56:11.742508 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.742369    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e03981-1083-4f6b-8004-317a2b6ed311-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-579321-predictor-5d7f57b548-wpthj\" (UID: \"09e03981-1083-4f6b-8004-317a2b6ed311\") " pod="kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj"
Sep 16 18:56:11.742508 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.742399    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kv6k\" (UniqueName: \"kubernetes.io/projected/09e03981-1083-4f6b-8004-317a2b6ed311-kube-api-access-2kv6k\") pod \"isvc-init-fail-579321-predictor-5d7f57b548-wpthj\" (UID: \"09e03981-1083-4f6b-8004-317a2b6ed311\") " pod="kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj"
Sep 16 18:56:11.742940 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.742916    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09e03981-1083-4f6b-8004-317a2b6ed311-kserve-provision-location\") pod \"isvc-init-fail-579321-predictor-5d7f57b548-wpthj\" (UID: \"09e03981-1083-4f6b-8004-317a2b6ed311\") " pod="kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj"
Sep 16 18:56:11.743082 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.743061    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e03981-1083-4f6b-8004-317a2b6ed311-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-579321-predictor-5d7f57b548-wpthj\" (UID: \"09e03981-1083-4f6b-8004-317a2b6ed311\") " pod="kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj"
Sep 16 18:56:11.743144 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.743061    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09e03981-1083-4f6b-8004-317a2b6ed311-cabundle-cert\") pod \"isvc-init-fail-579321-predictor-5d7f57b548-wpthj\" (UID: \"09e03981-1083-4f6b-8004-317a2b6ed311\") " pod="kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj"
Sep 16 18:56:11.744940 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.744925    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e03981-1083-4f6b-8004-317a2b6ed311-proxy-tls\") pod \"isvc-init-fail-579321-predictor-5d7f57b548-wpthj\" (UID: \"09e03981-1083-4f6b-8004-317a2b6ed311\") " pod="kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj"
Sep 16 18:56:11.751203 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.751179    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kv6k\" (UniqueName: \"kubernetes.io/projected/09e03981-1083-4f6b-8004-317a2b6ed311-kube-api-access-2kv6k\") pod \"isvc-init-fail-579321-predictor-5d7f57b548-wpthj\" (UID: \"09e03981-1083-4f6b-8004-317a2b6ed311\") " pod="kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj"
Sep 16 18:56:11.759944 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.759914    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj"
Sep 16 18:56:11.860106 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:11.860080    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"]
Sep 16 18:56:11.861831 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:56:11.861805    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63560122_fda1_4051_b4af_2b6bb701688d.slice/crio-670576c582f134311955dd2b3e3ca8a90e14656e558e7ff82fae08175e08594e WatchSource:0}: Error finding container 670576c582f134311955dd2b3e3ca8a90e14656e558e7ff82fae08175e08594e: Status 404 returned error can't find the container with id 670576c582f134311955dd2b3e3ca8a90e14656e558e7ff82fae08175e08594e
Sep 16 18:56:12.032417 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:12.032385    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj"]
Sep 16 18:56:12.034453 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:56:12.034427    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09e03981_1083_4f6b_8004_317a2b6ed311.slice/crio-9137d46bdb86f2f4ff62654fbd2da21b6a6065fbedd55c0860875ec62e0ff81e WatchSource:0}: Error finding container 9137d46bdb86f2f4ff62654fbd2da21b6a6065fbedd55c0860875ec62e0ff81e: Status 404 returned error can't find the container with id 9137d46bdb86f2f4ff62654fbd2da21b6a6065fbedd55c0860875ec62e0ff81e
Sep 16 18:56:12.346693 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:12.346603    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj" event={"ID":"09e03981-1083-4f6b-8004-317a2b6ed311","Type":"ContainerStarted","Data":"eb4f5146c4354ebb61e4629a4bb5244753670b2ed2ba8c9dd12cbfa886e7f081"}
Sep 16 18:56:12.346693 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:12.346673    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj" event={"ID":"09e03981-1083-4f6b-8004-317a2b6ed311","Type":"ContainerStarted","Data":"9137d46bdb86f2f4ff62654fbd2da21b6a6065fbedd55c0860875ec62e0ff81e"}
Sep 16 18:56:12.348595 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:12.348568    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" event={"ID":"63560122-fda1-4051-b4af-2b6bb701688d","Type":"ContainerStarted","Data":"adf81f2af068c9cd5722d450f81122ff15b26d69f8bd9ccfcd6c4d42032af7ec"}
Sep 16 18:56:12.348753 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:12.348622    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" event={"ID":"63560122-fda1-4051-b4af-2b6bb701688d","Type":"ContainerStarted","Data":"670576c582f134311955dd2b3e3ca8a90e14656e558e7ff82fae08175e08594e"}
Sep 16 18:56:16.369543 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:16.369512    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-579321-predictor-5d7f57b548-wpthj_09e03981-1083-4f6b-8004-317a2b6ed311/storage-initializer/0.log"
Sep 16 18:56:16.370053 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:16.369551    2554 generic.go:356] "Generic (PLEG): container finished" podID="09e03981-1083-4f6b-8004-317a2b6ed311" containerID="eb4f5146c4354ebb61e4629a4bb5244753670b2ed2ba8c9dd12cbfa886e7f081" exitCode=1
Sep 16 18:56:16.370053 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:16.369640    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj" event={"ID":"09e03981-1083-4f6b-8004-317a2b6ed311","Type":"ContainerDied","Data":"eb4f5146c4354ebb61e4629a4bb5244753670b2ed2ba8c9dd12cbfa886e7f081"}
Sep 16 18:56:16.371248 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:16.371229    2554 generic.go:356] "Generic (PLEG): container finished" podID="63560122-fda1-4051-b4af-2b6bb701688d" containerID="adf81f2af068c9cd5722d450f81122ff15b26d69f8bd9ccfcd6c4d42032af7ec" exitCode=0
Sep 16 18:56:16.371325 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:16.371287    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" event={"ID":"63560122-fda1-4051-b4af-2b6bb701688d","Type":"ContainerDied","Data":"adf81f2af068c9cd5722d450f81122ff15b26d69f8bd9ccfcd6c4d42032af7ec"}
Sep 16 18:56:17.399757 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:17.399708    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-579321-predictor-5d7f57b548-wpthj_09e03981-1083-4f6b-8004-317a2b6ed311/storage-initializer/0.log"
Sep 16 18:56:17.400226 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:17.399884    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj" event={"ID":"09e03981-1083-4f6b-8004-317a2b6ed311","Type":"ContainerStarted","Data":"a66050861dba203f3f80ecf802e68e199612ef7dc75339f5335a6defb6297fcf"}
Sep 16 18:56:20.960739 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:20.960329    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj"]
Sep 16 18:56:20.960739 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:20.960586    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj" podUID="09e03981-1083-4f6b-8004-317a2b6ed311" containerName="storage-initializer" containerID="cri-o://a66050861dba203f3f80ecf802e68e199612ef7dc75339f5335a6defb6297fcf" gracePeriod=30
Sep 16 18:56:22.444158 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:22.444124    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-579321-predictor-5d7f57b548-wpthj_09e03981-1083-4f6b-8004-317a2b6ed311/storage-initializer/1.log"
Sep 16 18:56:22.444677 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:22.444580    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-579321-predictor-5d7f57b548-wpthj_09e03981-1083-4f6b-8004-317a2b6ed311/storage-initializer/0.log"
Sep 16 18:56:22.444677 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:22.444616    2554 generic.go:356] "Generic (PLEG): container finished" podID="09e03981-1083-4f6b-8004-317a2b6ed311" containerID="a66050861dba203f3f80ecf802e68e199612ef7dc75339f5335a6defb6297fcf" exitCode=1
Sep 16 18:56:22.444843 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:22.444704    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj" event={"ID":"09e03981-1083-4f6b-8004-317a2b6ed311","Type":"ContainerDied","Data":"a66050861dba203f3f80ecf802e68e199612ef7dc75339f5335a6defb6297fcf"}
Sep 16 18:56:22.444843 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:22.444800    2554 scope.go:117] "RemoveContainer" containerID="eb4f5146c4354ebb61e4629a4bb5244753670b2ed2ba8c9dd12cbfa886e7f081"
Sep 16 18:56:25.247310 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:25.247286    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-579321-predictor-5d7f57b548-wpthj_09e03981-1083-4f6b-8004-317a2b6ed311/storage-initializer/1.log"
Sep 16 18:56:25.258768 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:25.258747    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj"
Sep 16 18:56:25.380052 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:25.380013    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kv6k\" (UniqueName: \"kubernetes.io/projected/09e03981-1083-4f6b-8004-317a2b6ed311-kube-api-access-2kv6k\") pod \"09e03981-1083-4f6b-8004-317a2b6ed311\" (UID: \"09e03981-1083-4f6b-8004-317a2b6ed311\") "
Sep 16 18:56:25.380230 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:25.380061    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e03981-1083-4f6b-8004-317a2b6ed311-proxy-tls\") pod \"09e03981-1083-4f6b-8004-317a2b6ed311\" (UID: \"09e03981-1083-4f6b-8004-317a2b6ed311\") "
Sep 16 18:56:25.380230 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:25.380139    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09e03981-1083-4f6b-8004-317a2b6ed311-cabundle-cert\") pod \"09e03981-1083-4f6b-8004-317a2b6ed311\" (UID: \"09e03981-1083-4f6b-8004-317a2b6ed311\") "
Sep 16 18:56:25.380230 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:25.380177    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e03981-1083-4f6b-8004-317a2b6ed311-kube-rbac-proxy-sar-config\") pod \"09e03981-1083-4f6b-8004-317a2b6ed311\" (UID: \"09e03981-1083-4f6b-8004-317a2b6ed311\") "
Sep 16 18:56:25.380230 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:25.380228    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09e03981-1083-4f6b-8004-317a2b6ed311-kserve-provision-location\") pod \"09e03981-1083-4f6b-8004-317a2b6ed311\" (UID: \"09e03981-1083-4f6b-8004-317a2b6ed311\") "
Sep 16 18:56:25.380749 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:25.380625    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09e03981-1083-4f6b-8004-317a2b6ed311-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "09e03981-1083-4f6b-8004-317a2b6ed311" (UID: "09e03981-1083-4f6b-8004-317a2b6ed311"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:56:25.380749 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:25.380633    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e03981-1083-4f6b-8004-317a2b6ed311-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "09e03981-1083-4f6b-8004-317a2b6ed311" (UID: "09e03981-1083-4f6b-8004-317a2b6ed311"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:56:25.380749 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:25.380629    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e03981-1083-4f6b-8004-317a2b6ed311-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "09e03981-1083-4f6b-8004-317a2b6ed311" (UID: "09e03981-1083-4f6b-8004-317a2b6ed311"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:56:25.382574 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:25.382536    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09e03981-1083-4f6b-8004-317a2b6ed311-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "09e03981-1083-4f6b-8004-317a2b6ed311" (UID: "09e03981-1083-4f6b-8004-317a2b6ed311"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:56:25.382684 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:25.382641    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09e03981-1083-4f6b-8004-317a2b6ed311-kube-api-access-2kv6k" (OuterVolumeSpecName: "kube-api-access-2kv6k") pod "09e03981-1083-4f6b-8004-317a2b6ed311" (UID: "09e03981-1083-4f6b-8004-317a2b6ed311"). InnerVolumeSpecName "kube-api-access-2kv6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:56:25.463989 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:25.463960    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-579321-predictor-5d7f57b548-wpthj_09e03981-1083-4f6b-8004-317a2b6ed311/storage-initializer/1.log"
Sep 16 18:56:25.464475 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:25.464453    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj"
Sep 16 18:56:25.464475 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:25.464457    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj" event={"ID":"09e03981-1083-4f6b-8004-317a2b6ed311","Type":"ContainerDied","Data":"9137d46bdb86f2f4ff62654fbd2da21b6a6065fbedd55c0860875ec62e0ff81e"}
Sep 16 18:56:25.480955 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:25.480896    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/09e03981-1083-4f6b-8004-317a2b6ed311-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:56:25.480955 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:25.480923    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kv6k\" (UniqueName: \"kubernetes.io/projected/09e03981-1083-4f6b-8004-317a2b6ed311-kube-api-access-2kv6k\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:56:25.480955 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:25.480939    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/09e03981-1083-4f6b-8004-317a2b6ed311-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:56:25.480955 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:25.480952    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/09e03981-1083-4f6b-8004-317a2b6ed311-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:56:25.481144 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:25.480965    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/09e03981-1083-4f6b-8004-317a2b6ed311-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:56:25.503973 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:25.503948    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj"]
Sep 16 18:56:25.507639 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:25.507616    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-579321-predictor-5d7f57b548-wpthj"]
Sep 16 18:56:26.145168 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:26.145138    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"]
Sep 16 18:56:26.145634 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:26.145613    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09e03981-1083-4f6b-8004-317a2b6ed311" containerName="storage-initializer"
Sep 16 18:56:26.145634 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:26.145634    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="09e03981-1083-4f6b-8004-317a2b6ed311" containerName="storage-initializer"
Sep 16 18:56:26.145831 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:26.145739    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="09e03981-1083-4f6b-8004-317a2b6ed311" containerName="storage-initializer"
Sep 16 18:56:26.145831 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:26.145804    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09e03981-1083-4f6b-8004-317a2b6ed311" containerName="storage-initializer"
Sep 16 18:56:26.145831 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:26.145815    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="09e03981-1083-4f6b-8004-317a2b6ed311" containerName="storage-initializer"
Sep 16 18:56:26.145983 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:26.145885    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="09e03981-1083-4f6b-8004-317a2b6ed311" containerName="storage-initializer"
Sep 16 18:56:26.313383 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:26.313350    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"]
Sep 16 18:56:26.313864 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:26.313488    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"
Sep 16 18:56:26.316253 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:26.316218    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Sep 16 18:56:26.316377 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:26.316314    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Sep 16 18:56:26.388662 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:26.388634    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-slkrb\" (UID: \"7ec3e2b3-7e75-4e66-93c0-00301a5a4886\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"
Sep 16 18:56:26.388827 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:26.388701    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-slkrb\" (UID: \"7ec3e2b3-7e75-4e66-93c0-00301a5a4886\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"
Sep 16 18:56:26.388827 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:26.388743    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-slkrb\" (UID: \"7ec3e2b3-7e75-4e66-93c0-00301a5a4886\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"
Sep 16 18:56:26.388827 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:26.388770    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r76s\" (UniqueName: \"kubernetes.io/projected/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-kube-api-access-6r76s\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-slkrb\" (UID: \"7ec3e2b3-7e75-4e66-93c0-00301a5a4886\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"
Sep 16 18:56:26.388975 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:26.388924    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-slkrb\" (UID: \"7ec3e2b3-7e75-4e66-93c0-00301a5a4886\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"
Sep 16 18:56:26.445162 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:26.445099    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09e03981-1083-4f6b-8004-317a2b6ed311" path="/var/lib/kubelet/pods/09e03981-1083-4f6b-8004-317a2b6ed311/volumes"
Sep 16 18:56:26.489313 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:26.489286    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-slkrb\" (UID: \"7ec3e2b3-7e75-4e66-93c0-00301a5a4886\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"
Sep 16 18:56:26.489442 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:26.489324    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-slkrb\" (UID: \"7ec3e2b3-7e75-4e66-93c0-00301a5a4886\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"
Sep 16 18:56:26.489506 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:56:26.489437    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-serving-cert: secret "isvc-predictive-sklearn-predictor-serving-cert" not found
Sep 16 18:56:26.489562 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:56:26.489521    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-proxy-tls podName:7ec3e2b3-7e75-4e66-93c0-00301a5a4886 nodeName:}" failed. No retries permitted until 2026-09-16 18:56:26.989490275 +0000 UTC m=+697.065259936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-proxy-tls") pod "isvc-predictive-sklearn-predictor-5f849d69df-slkrb" (UID: "7ec3e2b3-7e75-4e66-93c0-00301a5a4886") : secret "isvc-predictive-sklearn-predictor-serving-cert" not found
Sep 16 18:56:26.489562 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:26.489545    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r76s\" (UniqueName: \"kubernetes.io/projected/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-kube-api-access-6r76s\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-slkrb\" (UID: \"7ec3e2b3-7e75-4e66-93c0-00301a5a4886\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"
Sep 16 18:56:26.489692 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:26.489676    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-slkrb\" (UID: \"7ec3e2b3-7e75-4e66-93c0-00301a5a4886\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"
Sep 16 18:56:26.489769 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:26.489745    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-slkrb\" (UID: \"7ec3e2b3-7e75-4e66-93c0-00301a5a4886\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"
Sep 16 18:56:26.490063 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:26.490031    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-slkrb\" (UID: \"7ec3e2b3-7e75-4e66-93c0-00301a5a4886\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"
Sep 16 18:56:26.490194 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:26.490174    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-slkrb\" (UID: \"7ec3e2b3-7e75-4e66-93c0-00301a5a4886\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"
Sep 16 18:56:26.490377 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:26.490361    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-slkrb\" (UID: \"7ec3e2b3-7e75-4e66-93c0-00301a5a4886\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"
Sep 16 18:56:26.498371 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:26.498353    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r76s\" (UniqueName: \"kubernetes.io/projected/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-kube-api-access-6r76s\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-slkrb\" (UID: \"7ec3e2b3-7e75-4e66-93c0-00301a5a4886\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"
Sep 16 18:56:26.994707 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:26.994662    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-slkrb\" (UID: \"7ec3e2b3-7e75-4e66-93c0-00301a5a4886\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"
Sep 16 18:56:26.997462 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:26.997434    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-slkrb\" (UID: \"7ec3e2b3-7e75-4e66-93c0-00301a5a4886\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"
Sep 16 18:56:27.225833 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:27.225798    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"
Sep 16 18:56:29.682524 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:29.682381    2554 scope.go:117] "RemoveContainer" containerID="a66050861dba203f3f80ecf802e68e199612ef7dc75339f5335a6defb6297fcf"
Sep 16 18:56:29.723869 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:29.723842    2554 scope.go:117] "RemoveContainer" containerID="eb4f5146c4354ebb61e4629a4bb5244753670b2ed2ba8c9dd12cbfa886e7f081"
Sep 16 18:56:29.724388 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:56:29.724308    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb4f5146c4354ebb61e4629a4bb5244753670b2ed2ba8c9dd12cbfa886e7f081\": container with ID starting with eb4f5146c4354ebb61e4629a4bb5244753670b2ed2ba8c9dd12cbfa886e7f081 not found: ID does not exist" containerID="eb4f5146c4354ebb61e4629a4bb5244753670b2ed2ba8c9dd12cbfa886e7f081"
Sep 16 18:56:29.724388 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:29.724352    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb4f5146c4354ebb61e4629a4bb5244753670b2ed2ba8c9dd12cbfa886e7f081"} err="failed to get container status \"eb4f5146c4354ebb61e4629a4bb5244753670b2ed2ba8c9dd12cbfa886e7f081\": rpc error: code = NotFound desc = could not find container \"eb4f5146c4354ebb61e4629a4bb5244753670b2ed2ba8c9dd12cbfa886e7f081\": container with ID starting with eb4f5146c4354ebb61e4629a4bb5244753670b2ed2ba8c9dd12cbfa886e7f081 not found: ID does not exist"
Sep 16 18:56:29.960604 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:29.960578    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"]
Sep 16 18:56:30.496958 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:30.496920    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" event={"ID":"63560122-fda1-4051-b4af-2b6bb701688d","Type":"ContainerStarted","Data":"1d25986486897253a7f38464d3a9323c34c13fbffde6573aea69548ce6e33d3a"}
Sep 16 18:56:30.499198 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:30.499049    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb" event={"ID":"7ec3e2b3-7e75-4e66-93c0-00301a5a4886","Type":"ContainerStarted","Data":"6213e8afdf708c1ee0f196c9914a610db42c62c72f0e4a049cb0777a4daad214"}
Sep 16 18:56:30.499198 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:30.499110    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb" event={"ID":"7ec3e2b3-7e75-4e66-93c0-00301a5a4886","Type":"ContainerStarted","Data":"6eb55674ddb2203b0f78a009471da487923da6815c8c07983ba0907a41b8bb3e"}
Sep 16 18:56:32.510903 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:32.510864    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" event={"ID":"63560122-fda1-4051-b4af-2b6bb701688d","Type":"ContainerStarted","Data":"fc3122fa4d3e6fde61c02a20ef2902f8506cda416aca4033218f794293ee09b2"}
Sep 16 18:56:34.524480 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:34.524444    2554 generic.go:356] "Generic (PLEG): container finished" podID="7ec3e2b3-7e75-4e66-93c0-00301a5a4886" containerID="6213e8afdf708c1ee0f196c9914a610db42c62c72f0e4a049cb0777a4daad214" exitCode=0
Sep 16 18:56:34.524980 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:34.524502    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb" event={"ID":"7ec3e2b3-7e75-4e66-93c0-00301a5a4886","Type":"ContainerDied","Data":"6213e8afdf708c1ee0f196c9914a610db42c62c72f0e4a049cb0777a4daad214"}
Sep 16 18:56:35.542896 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:35.542062    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" event={"ID":"63560122-fda1-4051-b4af-2b6bb701688d","Type":"ContainerStarted","Data":"1f16949a73a1ecf484429514476810dd06228367c5c22724c5c16815c6840c13"}
Sep 16 18:56:35.546079 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:35.545161    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"
Sep 16 18:56:35.546079 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:35.545224    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"
Sep 16 18:56:35.546079 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:35.545260    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"
Sep 16 18:56:35.546893 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:35.546503    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:56:35.546893 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:35.546804    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:56:36.553643 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:36.553088    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:56:36.553643 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:36.553576    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:56:37.559925 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:37.559519    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:56:37.560385 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:37.560210    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:56:37.567516 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:37.567347    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"
Sep 16 18:56:37.590621 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:37.590250    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podStartSLOduration=4.330924358 podStartE2EDuration="26.59023332s" totalImagesPullingTime="18.3432944s" totalInitContainerRuntime="3.916014562s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:35.578495775 +0000 UTC m=+705.654265408" watchObservedRunningTime="2026-09-16 18:56:37.59023332 +0000 UTC m=+707.666002953"
Sep 16 18:56:38.566641 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:38.566601    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:56:38.567139 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:38.567102    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:56:48.567087 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:48.566933    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:56:48.567619 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:48.567482    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:56:56.677576 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:56.677542    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb" event={"ID":"7ec3e2b3-7e75-4e66-93c0-00301a5a4886","Type":"ContainerStarted","Data":"71c7f831db355faafd631c4d4185509c48a611fe663d72ee845c84b0fa224810"}
Sep 16 18:56:56.678026 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:56.677602    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb" event={"ID":"7ec3e2b3-7e75-4e66-93c0-00301a5a4886","Type":"ContainerStarted","Data":"7555878ac922e1593742a685c00a4cec20a87542a2ad643ecc1c4b06b3d5e50d"}
Sep 16 18:56:56.678026 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:56.677769    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"
Sep 16 18:56:56.701112 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:56.701070    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb" podStartSLOduration=5.047662708 podStartE2EDuration="30.701054855s" totalImagesPullingTime="21.60056753s" totalInitContainerRuntime="4.052824617s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:56:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:56:56.699538135 +0000 UTC m=+726.775307766" watchObservedRunningTime="2026-09-16 18:56:56.701054855 +0000 UTC m=+726.776824554"
Sep 16 18:56:57.682834 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:57.682806    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"
Sep 16 18:56:57.684087 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:57.684056    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb" podUID="7ec3e2b3-7e75-4e66-93c0-00301a5a4886" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 18:56:58.566608 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:58.566569    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:56:58.567066 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:58.567040    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:56:58.687463 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:56:58.687421    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb" podUID="7ec3e2b3-7e75-4e66-93c0-00301a5a4886" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 18:57:03.691698 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:57:03.691668    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"
Sep 16 18:57:03.692190 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:57:03.692165    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb" podUID="7ec3e2b3-7e75-4e66-93c0-00301a5a4886" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 18:57:08.566868 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:57:08.566820    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:57:08.567312 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:57:08.567243    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:13.692164 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:57:13.692123    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb" podUID="7ec3e2b3-7e75-4e66-93c0-00301a5a4886" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 18:57:18.566154 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:57:18.566111    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:57:18.566681 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:57:18.566659    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:23.693048 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:57:23.693011    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb" podUID="7ec3e2b3-7e75-4e66-93c0-00301a5a4886" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 18:57:28.566446 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:57:28.566403    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:57:28.566972 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:57:28.566950    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:33.692794 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:57:33.692745    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb" podUID="7ec3e2b3-7e75-4e66-93c0-00301a5a4886" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 18:57:38.565954 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:57:38.565908    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:57:38.566420 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:57:38.566398    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:43.692991 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:57:43.692948    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb" podUID="7ec3e2b3-7e75-4e66-93c0-00301a5a4886" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 18:57:48.566082 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:57:48.566039    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:57:48.566575 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:57:48.566473    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:57:53.692902 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:57:53.692869    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb" podUID="7ec3e2b3-7e75-4e66-93c0-00301a5a4886" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 18:57:58.566895 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:57:58.566862    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"
Sep 16 18:57:58.567569 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:57:58.567547    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"
Sep 16 18:58:03.692861 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:03.692821    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb" podUID="7ec3e2b3-7e75-4e66-93c0-00301a5a4886" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Sep 16 18:58:06.054058 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:06.054022    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"]
Sep 16 18:58:06.054510 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:06.054326    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="kserve-container" containerID="cri-o://1d25986486897253a7f38464d3a9323c34c13fbffde6573aea69548ce6e33d3a" gracePeriod=30
Sep 16 18:58:06.054510 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:06.054401    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="kube-rbac-proxy" containerID="cri-o://fc3122fa4d3e6fde61c02a20ef2902f8506cda416aca4033218f794293ee09b2" gracePeriod=30
Sep 16 18:58:06.054510 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:06.054424    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="agent" containerID="cri-o://1f16949a73a1ecf484429514476810dd06228367c5c22724c5c16815c6840c13" gracePeriod=30
Sep 16 18:58:06.196448 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:06.196419    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"]
Sep 16 18:58:06.198843 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:06.198815    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"
Sep 16 18:58:06.200595 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:06.200572    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Sep 16 18:58:06.200740 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:06.200598    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Sep 16 18:58:06.210222 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:06.210202    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"]
Sep 16 18:58:06.279442 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:06.279408    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s7nd\" (UniqueName: \"kubernetes.io/projected/27d16f6a-7f81-4142-b920-723e5805a623-kube-api-access-9s7nd\") pod \"isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql\" (UID: \"27d16f6a-7f81-4142-b920-723e5805a623\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"
Sep 16 18:58:06.279617 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:06.279504    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27d16f6a-7f81-4142-b920-723e5805a623-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql\" (UID: \"27d16f6a-7f81-4142-b920-723e5805a623\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"
Sep 16 18:58:06.279617 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:06.279531    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27d16f6a-7f81-4142-b920-723e5805a623-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql\" (UID: \"27d16f6a-7f81-4142-b920-723e5805a623\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"
Sep 16 18:58:06.279617 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:06.279602    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27d16f6a-7f81-4142-b920-723e5805a623-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql\" (UID: \"27d16f6a-7f81-4142-b920-723e5805a623\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"
Sep 16 18:58:06.279824 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:06.279644    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27d16f6a-7f81-4142-b920-723e5805a623-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql\" (UID: \"27d16f6a-7f81-4142-b920-723e5805a623\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"
Sep 16 18:58:06.380681 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:06.380579    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27d16f6a-7f81-4142-b920-723e5805a623-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql\" (UID: \"27d16f6a-7f81-4142-b920-723e5805a623\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"
Sep 16 18:58:06.380681 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:06.380634    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27d16f6a-7f81-4142-b920-723e5805a623-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql\" (UID: \"27d16f6a-7f81-4142-b920-723e5805a623\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"
Sep 16 18:58:06.380681 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:06.380679    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27d16f6a-7f81-4142-b920-723e5805a623-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql\" (UID: \"27d16f6a-7f81-4142-b920-723e5805a623\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"
Sep 16 18:58:06.381005 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:06.380750    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27d16f6a-7f81-4142-b920-723e5805a623-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql\" (UID: \"27d16f6a-7f81-4142-b920-723e5805a623\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"
Sep 16 18:58:06.381005 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:58:06.380828    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-serving-cert: secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Sep 16 18:58:06.381005 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:06.380839    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s7nd\" (UniqueName: \"kubernetes.io/projected/27d16f6a-7f81-4142-b920-723e5805a623-kube-api-access-9s7nd\") pod \"isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql\" (UID: \"27d16f6a-7f81-4142-b920-723e5805a623\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"
Sep 16 18:58:06.381005 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:58:06.380919    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27d16f6a-7f81-4142-b920-723e5805a623-proxy-tls podName:27d16f6a-7f81-4142-b920-723e5805a623 nodeName:}" failed. No retries permitted until 2026-09-16 18:58:06.880895462 +0000 UTC m=+796.956665075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/27d16f6a-7f81-4142-b920-723e5805a623-proxy-tls") pod "isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" (UID: "27d16f6a-7f81-4142-b920-723e5805a623") : secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Sep 16 18:58:06.381190 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:06.381152    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27d16f6a-7f81-4142-b920-723e5805a623-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql\" (UID: \"27d16f6a-7f81-4142-b920-723e5805a623\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"
Sep 16 18:58:06.381456 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:06.381438    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27d16f6a-7f81-4142-b920-723e5805a623-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql\" (UID: \"27d16f6a-7f81-4142-b920-723e5805a623\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"
Sep 16 18:58:06.381493 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:06.381451    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27d16f6a-7f81-4142-b920-723e5805a623-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql\" (UID: \"27d16f6a-7f81-4142-b920-723e5805a623\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"
Sep 16 18:58:06.389706 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:06.389685    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s7nd\" (UniqueName: \"kubernetes.io/projected/27d16f6a-7f81-4142-b920-723e5805a623-kube-api-access-9s7nd\") pod \"isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql\" (UID: \"27d16f6a-7f81-4142-b920-723e5805a623\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"
Sep 16 18:58:06.885178 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:06.885142    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27d16f6a-7f81-4142-b920-723e5805a623-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql\" (UID: \"27d16f6a-7f81-4142-b920-723e5805a623\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"
Sep 16 18:58:06.887389 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:06.887369    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27d16f6a-7f81-4142-b920-723e5805a623-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql\" (UID: \"27d16f6a-7f81-4142-b920-723e5805a623\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"
Sep 16 18:58:07.009338 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:07.009312    2554 generic.go:356] "Generic (PLEG): container finished" podID="63560122-fda1-4051-b4af-2b6bb701688d" containerID="fc3122fa4d3e6fde61c02a20ef2902f8506cda416aca4033218f794293ee09b2" exitCode=2
Sep 16 18:58:07.009470 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:07.009390    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" event={"ID":"63560122-fda1-4051-b4af-2b6bb701688d","Type":"ContainerDied","Data":"fc3122fa4d3e6fde61c02a20ef2902f8506cda416aca4033218f794293ee09b2"}
Sep 16 18:58:07.108792 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:07.108765    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"
Sep 16 18:58:07.384582 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:07.384556    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"]
Sep 16 18:58:07.386270 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:58:07.386235    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d16f6a_7f81_4142_b920_723e5805a623.slice/crio-763fb517177bfc0b16d8aef63a0420da8d4d6dba7390553d6bfff07f3141e134 WatchSource:0}: Error finding container 763fb517177bfc0b16d8aef63a0420da8d4d6dba7390553d6bfff07f3141e134: Status 404 returned error can't find the container with id 763fb517177bfc0b16d8aef63a0420da8d4d6dba7390553d6bfff07f3141e134
Sep 16 18:58:07.560253 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:07.560219    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 16 18:58:08.017203 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:08.017164    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" event={"ID":"27d16f6a-7f81-4142-b920-723e5805a623","Type":"ContainerStarted","Data":"86b9992d1428ad4ad86ace0fd4722c680fc368f3ac51c8a0dac2d2335fa05bc1"}
Sep 16 18:58:08.017361 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:08.017248    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" event={"ID":"27d16f6a-7f81-4142-b920-723e5805a623","Type":"ContainerStarted","Data":"763fb517177bfc0b16d8aef63a0420da8d4d6dba7390553d6bfff07f3141e134"}
Sep 16 18:58:08.566272 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:08.566236    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:58:08.568902 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:08.568871    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:11.033862 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:11.033827    2554 generic.go:356] "Generic (PLEG): container finished" podID="63560122-fda1-4051-b4af-2b6bb701688d" containerID="1d25986486897253a7f38464d3a9323c34c13fbffde6573aea69548ce6e33d3a" exitCode=0
Sep 16 18:58:11.034244 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:11.033905    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" event={"ID":"63560122-fda1-4051-b4af-2b6bb701688d","Type":"ContainerDied","Data":"1d25986486897253a7f38464d3a9323c34c13fbffde6573aea69548ce6e33d3a"}
Sep 16 18:58:12.039794 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:12.039758    2554 generic.go:356] "Generic (PLEG): container finished" podID="27d16f6a-7f81-4142-b920-723e5805a623" containerID="86b9992d1428ad4ad86ace0fd4722c680fc368f3ac51c8a0dac2d2335fa05bc1" exitCode=0
Sep 16 18:58:12.040198 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:12.039831    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" event={"ID":"27d16f6a-7f81-4142-b920-723e5805a623","Type":"ContainerDied","Data":"86b9992d1428ad4ad86ace0fd4722c680fc368f3ac51c8a0dac2d2335fa05bc1"}
Sep 16 18:58:12.559959 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:12.559921    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 16 18:58:13.047592 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:13.047560    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" event={"ID":"27d16f6a-7f81-4142-b920-723e5805a623","Type":"ContainerStarted","Data":"9a96dd75d7d7ace482d072d4b4d2d71df085e23bbc726f1e65d8b50f41086b05"}
Sep 16 18:58:13.047998 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:13.047623    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" event={"ID":"27d16f6a-7f81-4142-b920-723e5805a623","Type":"ContainerStarted","Data":"1092781c22526f598cec1af475601c77a01da143fa5b44c900760c5b37432134"}
Sep 16 18:58:13.047998 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:13.047650    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" event={"ID":"27d16f6a-7f81-4142-b920-723e5805a623","Type":"ContainerStarted","Data":"1da8a30e5b9e972a1c531de558c20d3de9f5de13972d2806f5988aff7cbd3211"}
Sep 16 18:58:13.048110 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:13.047996    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"
Sep 16 18:58:13.048110 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:13.048033    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"
Sep 16 18:58:13.049426 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:13.049401    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Sep 16 18:58:13.070422 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:13.070366    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podStartSLOduration=3.246006319 podStartE2EDuration="7.070351163s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.824344844s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:13.06825241 +0000 UTC m=+803.144022041" watchObservedRunningTime="2026-09-16 18:58:13.070351163 +0000 UTC m=+803.146120795"
Sep 16 18:58:13.692889 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:13.692860    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"
Sep 16 18:58:14.052403 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:14.052362    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Sep 16 18:58:14.052823 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:14.052501    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"
Sep 16 18:58:14.053681 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:14.053655    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:15.057030 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:15.056987    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Sep 16 18:58:15.057516 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:15.057457    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:16.232843 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:16.232815    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"]
Sep 16 18:58:16.233248 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:16.233093    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb" podUID="7ec3e2b3-7e75-4e66-93c0-00301a5a4886" containerName="kserve-container" containerID="cri-o://7555878ac922e1593742a685c00a4cec20a87542a2ad643ecc1c4b06b3d5e50d" gracePeriod=30
Sep 16 18:58:16.233248 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:16.233152    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb" podUID="7ec3e2b3-7e75-4e66-93c0-00301a5a4886" containerName="kube-rbac-proxy" containerID="cri-o://71c7f831db355faafd631c4d4185509c48a611fe663d72ee845c84b0fa224810" gracePeriod=30
Sep 16 18:58:17.070016 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:17.069982    2554 generic.go:356] "Generic (PLEG): container finished" podID="7ec3e2b3-7e75-4e66-93c0-00301a5a4886" containerID="71c7f831db355faafd631c4d4185509c48a611fe663d72ee845c84b0fa224810" exitCode=2
Sep 16 18:58:17.070193 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:17.070059    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb" event={"ID":"7ec3e2b3-7e75-4e66-93c0-00301a5a4886","Type":"ContainerDied","Data":"71c7f831db355faafd631c4d4185509c48a611fe663d72ee845c84b0fa224810"}
Sep 16 18:58:17.559681 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:17.559633    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 16 18:58:17.560137 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:17.559791    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"
Sep 16 18:58:18.565940 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:18.565903    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:58:18.568497 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:18.568477    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:18.688184 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:18.688147    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb" podUID="7ec3e2b3-7e75-4e66-93c0-00301a5a4886" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Sep 16 18:58:20.062022 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:20.061990    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"
Sep 16 18:58:20.062530 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:20.062500    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Sep 16 18:58:20.062874 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:20.062852    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:20.677678 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:20.677651    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"
Sep 16 18:58:20.803338 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:20.803312    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-kube-rbac-proxy-sar-config\") pod \"7ec3e2b3-7e75-4e66-93c0-00301a5a4886\" (UID: \"7ec3e2b3-7e75-4e66-93c0-00301a5a4886\") "
Sep 16 18:58:20.803510 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:20.803364    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6r76s\" (UniqueName: \"kubernetes.io/projected/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-kube-api-access-6r76s\") pod \"7ec3e2b3-7e75-4e66-93c0-00301a5a4886\" (UID: \"7ec3e2b3-7e75-4e66-93c0-00301a5a4886\") "
Sep 16 18:58:20.803510 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:20.803442    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-cabundle-cert\") pod \"7ec3e2b3-7e75-4e66-93c0-00301a5a4886\" (UID: \"7ec3e2b3-7e75-4e66-93c0-00301a5a4886\") "
Sep 16 18:58:20.803510 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:20.803491    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-proxy-tls\") pod \"7ec3e2b3-7e75-4e66-93c0-00301a5a4886\" (UID: \"7ec3e2b3-7e75-4e66-93c0-00301a5a4886\") "
Sep 16 18:58:20.803617 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:20.803517    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-kserve-provision-location\") pod \"7ec3e2b3-7e75-4e66-93c0-00301a5a4886\" (UID: \"7ec3e2b3-7e75-4e66-93c0-00301a5a4886\") "
Sep 16 18:58:20.803823 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:20.803800    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7ec3e2b3-7e75-4e66-93c0-00301a5a4886" (UID: "7ec3e2b3-7e75-4e66-93c0-00301a5a4886"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:20.803940 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:20.803914    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7ec3e2b3-7e75-4e66-93c0-00301a5a4886" (UID: "7ec3e2b3-7e75-4e66-93c0-00301a5a4886"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:20.803989 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:20.803920    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7ec3e2b3-7e75-4e66-93c0-00301a5a4886" (UID: "7ec3e2b3-7e75-4e66-93c0-00301a5a4886"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:20.805586 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:20.805561    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-kube-api-access-6r76s" (OuterVolumeSpecName: "kube-api-access-6r76s") pod "7ec3e2b3-7e75-4e66-93c0-00301a5a4886" (UID: "7ec3e2b3-7e75-4e66-93c0-00301a5a4886"). InnerVolumeSpecName "kube-api-access-6r76s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:20.805586 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:20.805579    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7ec3e2b3-7e75-4e66-93c0-00301a5a4886" (UID: "7ec3e2b3-7e75-4e66-93c0-00301a5a4886"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:20.905017 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:20.904984    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:58:20.905017 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:20.905012    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r76s\" (UniqueName: \"kubernetes.io/projected/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-kube-api-access-6r76s\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:58:20.905017 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:20.905023    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:58:20.905238 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:20.905033    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:58:20.905238 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:20.905041    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ec3e2b3-7e75-4e66-93c0-00301a5a4886-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:58:21.091638 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.091545    2554 generic.go:356] "Generic (PLEG): container finished" podID="7ec3e2b3-7e75-4e66-93c0-00301a5a4886" containerID="7555878ac922e1593742a685c00a4cec20a87542a2ad643ecc1c4b06b3d5e50d" exitCode=0
Sep 16 18:58:21.091638 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.091585    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb" event={"ID":"7ec3e2b3-7e75-4e66-93c0-00301a5a4886","Type":"ContainerDied","Data":"7555878ac922e1593742a685c00a4cec20a87542a2ad643ecc1c4b06b3d5e50d"}
Sep 16 18:58:21.091638 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.091604    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"
Sep 16 18:58:21.092190 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.091645    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb" event={"ID":"7ec3e2b3-7e75-4e66-93c0-00301a5a4886","Type":"ContainerDied","Data":"6eb55674ddb2203b0f78a009471da487923da6815c8c07983ba0907a41b8bb3e"}
Sep 16 18:58:21.092190 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.091678    2554 scope.go:117] "RemoveContainer" containerID="71c7f831db355faafd631c4d4185509c48a611fe663d72ee845c84b0fa224810"
Sep 16 18:58:21.107187 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.107168    2554 scope.go:117] "RemoveContainer" containerID="7555878ac922e1593742a685c00a4cec20a87542a2ad643ecc1c4b06b3d5e50d"
Sep 16 18:58:21.114864 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.114842    2554 scope.go:117] "RemoveContainer" containerID="6213e8afdf708c1ee0f196c9914a610db42c62c72f0e4a049cb0777a4daad214"
Sep 16 18:58:21.120382 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.120361    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"]
Sep 16 18:58:21.122579 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.122559    2554 scope.go:117] "RemoveContainer" containerID="71c7f831db355faafd631c4d4185509c48a611fe663d72ee845c84b0fa224810"
Sep 16 18:58:21.122863 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:58:21.122841    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71c7f831db355faafd631c4d4185509c48a611fe663d72ee845c84b0fa224810\": container with ID starting with 71c7f831db355faafd631c4d4185509c48a611fe663d72ee845c84b0fa224810 not found: ID does not exist" containerID="71c7f831db355faafd631c4d4185509c48a611fe663d72ee845c84b0fa224810"
Sep 16 18:58:21.122934 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.122871    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71c7f831db355faafd631c4d4185509c48a611fe663d72ee845c84b0fa224810"} err="failed to get container status \"71c7f831db355faafd631c4d4185509c48a611fe663d72ee845c84b0fa224810\": rpc error: code = NotFound desc = could not find container \"71c7f831db355faafd631c4d4185509c48a611fe663d72ee845c84b0fa224810\": container with ID starting with 71c7f831db355faafd631c4d4185509c48a611fe663d72ee845c84b0fa224810 not found: ID does not exist"
Sep 16 18:58:21.122934 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.122887    2554 scope.go:117] "RemoveContainer" containerID="7555878ac922e1593742a685c00a4cec20a87542a2ad643ecc1c4b06b3d5e50d"
Sep 16 18:58:21.123247 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:58:21.123223    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7555878ac922e1593742a685c00a4cec20a87542a2ad643ecc1c4b06b3d5e50d\": container with ID starting with 7555878ac922e1593742a685c00a4cec20a87542a2ad643ecc1c4b06b3d5e50d not found: ID does not exist" containerID="7555878ac922e1593742a685c00a4cec20a87542a2ad643ecc1c4b06b3d5e50d"
Sep 16 18:58:21.123327 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.123251    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7555878ac922e1593742a685c00a4cec20a87542a2ad643ecc1c4b06b3d5e50d"} err="failed to get container status \"7555878ac922e1593742a685c00a4cec20a87542a2ad643ecc1c4b06b3d5e50d\": rpc error: code = NotFound desc = could not find container \"7555878ac922e1593742a685c00a4cec20a87542a2ad643ecc1c4b06b3d5e50d\": container with ID starting with 7555878ac922e1593742a685c00a4cec20a87542a2ad643ecc1c4b06b3d5e50d not found: ID does not exist"
Sep 16 18:58:21.123327 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.123271    2554 scope.go:117] "RemoveContainer" containerID="6213e8afdf708c1ee0f196c9914a610db42c62c72f0e4a049cb0777a4daad214"
Sep 16 18:58:21.123605 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:58:21.123583    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6213e8afdf708c1ee0f196c9914a610db42c62c72f0e4a049cb0777a4daad214\": container with ID starting with 6213e8afdf708c1ee0f196c9914a610db42c62c72f0e4a049cb0777a4daad214 not found: ID does not exist" containerID="6213e8afdf708c1ee0f196c9914a610db42c62c72f0e4a049cb0777a4daad214"
Sep 16 18:58:21.123692 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.123607    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6213e8afdf708c1ee0f196c9914a610db42c62c72f0e4a049cb0777a4daad214"} err="failed to get container status \"6213e8afdf708c1ee0f196c9914a610db42c62c72f0e4a049cb0777a4daad214\": rpc error: code = NotFound desc = could not find container \"6213e8afdf708c1ee0f196c9914a610db42c62c72f0e4a049cb0777a4daad214\": container with ID starting with 6213e8afdf708c1ee0f196c9914a610db42c62c72f0e4a049cb0777a4daad214 not found: ID does not exist"
Sep 16 18:58:21.125299 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.125270    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-slkrb"]
Sep 16 18:58:21.418608 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.418526    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7"]
Sep 16 18:58:21.418965 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.418951    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ec3e2b3-7e75-4e66-93c0-00301a5a4886" containerName="storage-initializer"
Sep 16 18:58:21.419012 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.418967    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ec3e2b3-7e75-4e66-93c0-00301a5a4886" containerName="storage-initializer"
Sep 16 18:58:21.419012 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.418986    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ec3e2b3-7e75-4e66-93c0-00301a5a4886" containerName="kserve-container"
Sep 16 18:58:21.419012 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.418991    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ec3e2b3-7e75-4e66-93c0-00301a5a4886" containerName="kserve-container"
Sep 16 18:58:21.419012 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.418997    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ec3e2b3-7e75-4e66-93c0-00301a5a4886" containerName="kube-rbac-proxy"
Sep 16 18:58:21.419012 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.419003    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ec3e2b3-7e75-4e66-93c0-00301a5a4886" containerName="kube-rbac-proxy"
Sep 16 18:58:21.419222 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.419066    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ec3e2b3-7e75-4e66-93c0-00301a5a4886" containerName="kube-rbac-proxy"
Sep 16 18:58:21.419222 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.419073    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ec3e2b3-7e75-4e66-93c0-00301a5a4886" containerName="kserve-container"
Sep 16 18:58:21.423937 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.423920    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7"
Sep 16 18:58:21.425969 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.425949    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Sep 16 18:58:21.426091 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.426072    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Sep 16 18:58:21.437011 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.436985    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7"]
Sep 16 18:58:21.611690 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.611656    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvmgm\" (UniqueName: \"kubernetes.io/projected/ec4a1895-4ab7-4302-bbf5-4b90a0779228-kube-api-access-wvmgm\") pod \"isvc-predictive-xgboost-predictor-695f978978-wz5l7\" (UID: \"ec4a1895-4ab7-4302-bbf5-4b90a0779228\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7"
Sep 16 18:58:21.611869 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.611697    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec4a1895-4ab7-4302-bbf5-4b90a0779228-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-wz5l7\" (UID: \"ec4a1895-4ab7-4302-bbf5-4b90a0779228\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7"
Sep 16 18:58:21.611869 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.611751    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec4a1895-4ab7-4302-bbf5-4b90a0779228-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-wz5l7\" (UID: \"ec4a1895-4ab7-4302-bbf5-4b90a0779228\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7"
Sep 16 18:58:21.611869 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.611800    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec4a1895-4ab7-4302-bbf5-4b90a0779228-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-wz5l7\" (UID: \"ec4a1895-4ab7-4302-bbf5-4b90a0779228\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7"
Sep 16 18:58:21.611869 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.611830    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec4a1895-4ab7-4302-bbf5-4b90a0779228-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-wz5l7\" (UID: \"ec4a1895-4ab7-4302-bbf5-4b90a0779228\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7"
Sep 16 18:58:21.713157 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.713066    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec4a1895-4ab7-4302-bbf5-4b90a0779228-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-wz5l7\" (UID: \"ec4a1895-4ab7-4302-bbf5-4b90a0779228\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7"
Sep 16 18:58:21.713157 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.713113    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec4a1895-4ab7-4302-bbf5-4b90a0779228-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-wz5l7\" (UID: \"ec4a1895-4ab7-4302-bbf5-4b90a0779228\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7"
Sep 16 18:58:21.713157 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.713156    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvmgm\" (UniqueName: \"kubernetes.io/projected/ec4a1895-4ab7-4302-bbf5-4b90a0779228-kube-api-access-wvmgm\") pod \"isvc-predictive-xgboost-predictor-695f978978-wz5l7\" (UID: \"ec4a1895-4ab7-4302-bbf5-4b90a0779228\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7"
Sep 16 18:58:21.713450 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.713176    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec4a1895-4ab7-4302-bbf5-4b90a0779228-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-wz5l7\" (UID: \"ec4a1895-4ab7-4302-bbf5-4b90a0779228\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7"
Sep 16 18:58:21.713450 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.713200    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec4a1895-4ab7-4302-bbf5-4b90a0779228-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-wz5l7\" (UID: \"ec4a1895-4ab7-4302-bbf5-4b90a0779228\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7"
Sep 16 18:58:21.713612 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.713593    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec4a1895-4ab7-4302-bbf5-4b90a0779228-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-wz5l7\" (UID: \"ec4a1895-4ab7-4302-bbf5-4b90a0779228\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7"
Sep 16 18:58:21.713893 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.713870    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec4a1895-4ab7-4302-bbf5-4b90a0779228-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-wz5l7\" (UID: \"ec4a1895-4ab7-4302-bbf5-4b90a0779228\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7"
Sep 16 18:58:21.713893 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.713887    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec4a1895-4ab7-4302-bbf5-4b90a0779228-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-wz5l7\" (UID: \"ec4a1895-4ab7-4302-bbf5-4b90a0779228\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7"
Sep 16 18:58:21.715779 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.715760    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec4a1895-4ab7-4302-bbf5-4b90a0779228-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-wz5l7\" (UID: \"ec4a1895-4ab7-4302-bbf5-4b90a0779228\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7"
Sep 16 18:58:21.722197 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.722178    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvmgm\" (UniqueName: \"kubernetes.io/projected/ec4a1895-4ab7-4302-bbf5-4b90a0779228-kube-api-access-wvmgm\") pod \"isvc-predictive-xgboost-predictor-695f978978-wz5l7\" (UID: \"ec4a1895-4ab7-4302-bbf5-4b90a0779228\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7"
Sep 16 18:58:21.733650 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:21.733628    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7"
Sep 16 18:58:22.015504 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:22.015470    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7"]
Sep 16 18:58:22.018377 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:58:22.018350    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4a1895_4ab7_4302_bbf5_4b90a0779228.slice/crio-b19692c56d647e95e0518fc0bd46a36153d2f0402ebf5ab8e5dfa89630c0eee3 WatchSource:0}: Error finding container b19692c56d647e95e0518fc0bd46a36153d2f0402ebf5ab8e5dfa89630c0eee3: Status 404 returned error can't find the container with id b19692c56d647e95e0518fc0bd46a36153d2f0402ebf5ab8e5dfa89630c0eee3
Sep 16 18:58:22.099175 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:22.099148    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7" event={"ID":"ec4a1895-4ab7-4302-bbf5-4b90a0779228","Type":"ContainerStarted","Data":"34de31f5e2928d3b3116fda6b640cf4fcc939a8d7a32a3df23acc0c567fd8aae"}
Sep 16 18:58:22.099572 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:22.099212    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7" event={"ID":"ec4a1895-4ab7-4302-bbf5-4b90a0779228","Type":"ContainerStarted","Data":"b19692c56d647e95e0518fc0bd46a36153d2f0402ebf5ab8e5dfa89630c0eee3"}
Sep 16 18:58:22.445917 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:22.445838    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ec3e2b3-7e75-4e66-93c0-00301a5a4886" path="/var/lib/kubelet/pods/7ec3e2b3-7e75-4e66-93c0-00301a5a4886/volumes"
Sep 16 18:58:22.560275 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:22.560227    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 16 18:58:26.122317 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:26.122277    2554 generic.go:356] "Generic (PLEG): container finished" podID="ec4a1895-4ab7-4302-bbf5-4b90a0779228" containerID="34de31f5e2928d3b3116fda6b640cf4fcc939a8d7a32a3df23acc0c567fd8aae" exitCode=0
Sep 16 18:58:26.122896 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:26.122321    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7" event={"ID":"ec4a1895-4ab7-4302-bbf5-4b90a0779228","Type":"ContainerDied","Data":"34de31f5e2928d3b3116fda6b640cf4fcc939a8d7a32a3df23acc0c567fd8aae"}
Sep 16 18:58:27.129791 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:27.129755    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7" event={"ID":"ec4a1895-4ab7-4302-bbf5-4b90a0779228","Type":"ContainerStarted","Data":"ea317eba517642a4b1f724b1a4117962611a6e266ca13c3f1780c4ccd096a0d6"}
Sep 16 18:58:27.130288 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:27.129828    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7" event={"ID":"ec4a1895-4ab7-4302-bbf5-4b90a0779228","Type":"ContainerStarted","Data":"8d895b69fd22ebea7906b56915c42e4e62e994111c611428d0fe055e856f2e21"}
Sep 16 18:58:27.130288 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:27.130043    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7"
Sep 16 18:58:27.150352 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:27.150294    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7" podStartSLOduration=2.5634330370000002 podStartE2EDuration="6.150276327s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.58684329s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 18:58:27.147962168 +0000 UTC m=+817.223731800" watchObservedRunningTime="2026-09-16 18:58:27.150276327 +0000 UTC m=+817.226045963"
Sep 16 18:58:27.560146 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:27.560106    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 16 18:58:28.135060 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:28.135025    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7"
Sep 16 18:58:28.136459 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:28.136426    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7" podUID="ec4a1895-4ab7-4302-bbf5-4b90a0779228" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:58:28.566033 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:28.565990    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Sep 16 18:58:28.566226 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:28.566117    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"
Sep 16 18:58:28.568699 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:28.568677    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:28.568829 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:28.568785    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"
Sep 16 18:58:29.140372 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:29.140331    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7" podUID="ec4a1895-4ab7-4302-bbf5-4b90a0779228" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:58:30.062805 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:30.062767    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Sep 16 18:58:30.063311 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:30.063287    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:32.559432 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:32.559393    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Sep 16 18:58:34.145234 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:34.145148    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7"
Sep 16 18:58:34.145869 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:34.145842    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7" podUID="ec4a1895-4ab7-4302-bbf5-4b90a0779228" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:58:36.183706 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:36.183668    2554 generic.go:356] "Generic (PLEG): container finished" podID="63560122-fda1-4051-b4af-2b6bb701688d" containerID="1f16949a73a1ecf484429514476810dd06228367c5c22724c5c16815c6840c13" exitCode=0
Sep 16 18:58:36.184197 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:36.183759    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" event={"ID":"63560122-fda1-4051-b4af-2b6bb701688d","Type":"ContainerDied","Data":"1f16949a73a1ecf484429514476810dd06228367c5c22724c5c16815c6840c13"}
Sep 16 18:58:36.204806 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:36.204770    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"
Sep 16 18:58:36.244652 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:36.244625    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63560122-fda1-4051-b4af-2b6bb701688d-kube-rbac-proxy-sar-config\") pod \"63560122-fda1-4051-b4af-2b6bb701688d\" (UID: \"63560122-fda1-4051-b4af-2b6bb701688d\") "
Sep 16 18:58:36.244853 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:36.244690    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63560122-fda1-4051-b4af-2b6bb701688d-kserve-provision-location\") pod \"63560122-fda1-4051-b4af-2b6bb701688d\" (UID: \"63560122-fda1-4051-b4af-2b6bb701688d\") "
Sep 16 18:58:36.244853 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:36.244756    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcps9\" (UniqueName: \"kubernetes.io/projected/63560122-fda1-4051-b4af-2b6bb701688d-kube-api-access-bcps9\") pod \"63560122-fda1-4051-b4af-2b6bb701688d\" (UID: \"63560122-fda1-4051-b4af-2b6bb701688d\") "
Sep 16 18:58:36.244853 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:36.244787    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63560122-fda1-4051-b4af-2b6bb701688d-cabundle-cert\") pod \"63560122-fda1-4051-b4af-2b6bb701688d\" (UID: \"63560122-fda1-4051-b4af-2b6bb701688d\") "
Sep 16 18:58:36.245068 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:36.244894    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63560122-fda1-4051-b4af-2b6bb701688d-proxy-tls\") pod \"63560122-fda1-4051-b4af-2b6bb701688d\" (UID: \"63560122-fda1-4051-b4af-2b6bb701688d\") "
Sep 16 18:58:36.245068 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:36.244981    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63560122-fda1-4051-b4af-2b6bb701688d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "63560122-fda1-4051-b4af-2b6bb701688d" (UID: "63560122-fda1-4051-b4af-2b6bb701688d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:36.245068 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:36.245021    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63560122-fda1-4051-b4af-2b6bb701688d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "63560122-fda1-4051-b4af-2b6bb701688d" (UID: "63560122-fda1-4051-b4af-2b6bb701688d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:58:36.245294 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:36.245272    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63560122-fda1-4051-b4af-2b6bb701688d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "63560122-fda1-4051-b4af-2b6bb701688d" (UID: "63560122-fda1-4051-b4af-2b6bb701688d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:58:36.245361 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:36.245278    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/63560122-fda1-4051-b4af-2b6bb701688d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:58:36.245361 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:36.245315    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/63560122-fda1-4051-b4af-2b6bb701688d-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:58:36.246916 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:36.246880    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63560122-fda1-4051-b4af-2b6bb701688d-kube-api-access-bcps9" (OuterVolumeSpecName: "kube-api-access-bcps9") pod "63560122-fda1-4051-b4af-2b6bb701688d" (UID: "63560122-fda1-4051-b4af-2b6bb701688d"). InnerVolumeSpecName "kube-api-access-bcps9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:58:36.247043 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:36.246944    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63560122-fda1-4051-b4af-2b6bb701688d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "63560122-fda1-4051-b4af-2b6bb701688d" (UID: "63560122-fda1-4051-b4af-2b6bb701688d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:58:36.346542 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:36.346453    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/63560122-fda1-4051-b4af-2b6bb701688d-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:58:36.346542 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:36.346487    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcps9\" (UniqueName: \"kubernetes.io/projected/63560122-fda1-4051-b4af-2b6bb701688d-kube-api-access-bcps9\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:58:36.346542 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:36.346496    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/63560122-fda1-4051-b4af-2b6bb701688d-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:58:37.190711 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:37.190676    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"
Sep 16 18:58:37.190711 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:37.190686    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7" event={"ID":"63560122-fda1-4051-b4af-2b6bb701688d","Type":"ContainerDied","Data":"670576c582f134311955dd2b3e3ca8a90e14656e558e7ff82fae08175e08594e"}
Sep 16 18:58:37.191197 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:37.190783    2554 scope.go:117] "RemoveContainer" containerID="1f16949a73a1ecf484429514476810dd06228367c5c22724c5c16815c6840c13"
Sep 16 18:58:37.199204 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:37.199185    2554 scope.go:117] "RemoveContainer" containerID="fc3122fa4d3e6fde61c02a20ef2902f8506cda416aca4033218f794293ee09b2"
Sep 16 18:58:37.206505 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:37.206488    2554 scope.go:117] "RemoveContainer" containerID="1d25986486897253a7f38464d3a9323c34c13fbffde6573aea69548ce6e33d3a"
Sep 16 18:58:37.211021 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:37.210999    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"]
Sep 16 18:58:37.216086 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:37.216066    2554 scope.go:117] "RemoveContainer" containerID="adf81f2af068c9cd5722d450f81122ff15b26d69f8bd9ccfcd6c4d42032af7ec"
Sep 16 18:58:37.219592 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:37.219571    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-74694d5594-2pqr7"]
Sep 16 18:58:38.442979 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:38.442946    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63560122-fda1-4051-b4af-2b6bb701688d" path="/var/lib/kubelet/pods/63560122-fda1-4051-b4af-2b6bb701688d/volumes"
Sep 16 18:58:40.062926 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:40.062887    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Sep 16 18:58:40.063382 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:40.063356    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:44.146082 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:44.146041    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7" podUID="ec4a1895-4ab7-4302-bbf5-4b90a0779228" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:58:50.062512 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:50.062462    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Sep 16 18:58:50.062907 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:50.062872    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:58:54.146076 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:58:54.146035    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7" podUID="ec4a1895-4ab7-4302-bbf5-4b90a0779228" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:59:00.063358 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:00.063307    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Sep 16 18:59:00.063931 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:00.063772    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:04.146883 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:04.146842    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7" podUID="ec4a1895-4ab7-4302-bbf5-4b90a0779228" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:59:10.062526 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:10.062486    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Sep 16 18:59:10.062939 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:10.062843    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:14.146844 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:14.146802    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7" podUID="ec4a1895-4ab7-4302-bbf5-4b90a0779228" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:59:20.062681 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:20.062632    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Sep 16 18:59:20.063271 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:20.063169    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:24.146099 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:24.146060    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7" podUID="ec4a1895-4ab7-4302-bbf5-4b90a0779228" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:59:30.063825 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:30.063794    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"
Sep 16 18:59:30.064242 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:30.063889    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"
Sep 16 18:59:34.146252 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:34.146218    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7" podUID="ec4a1895-4ab7-4302-bbf5-4b90a0779228" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:59:41.252322 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:41.252290    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"]
Sep 16 18:59:41.252776 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:41.252547    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="kserve-container" containerID="cri-o://1da8a30e5b9e972a1c531de558c20d3de9f5de13972d2806f5988aff7cbd3211" gracePeriod=30
Sep 16 18:59:41.252776 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:41.252575    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="agent" containerID="cri-o://9a96dd75d7d7ace482d072d4b4d2d71df085e23bbc726f1e65d8b50f41086b05" gracePeriod=30
Sep 16 18:59:41.252776 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:41.252586    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="kube-rbac-proxy" containerID="cri-o://1092781c22526f598cec1af475601c77a01da143fa5b44c900760c5b37432134" gracePeriod=30
Sep 16 18:59:41.503065 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:41.502976    2554 generic.go:356] "Generic (PLEG): container finished" podID="27d16f6a-7f81-4142-b920-723e5805a623" containerID="1092781c22526f598cec1af475601c77a01da143fa5b44c900760c5b37432134" exitCode=2
Sep 16 18:59:41.503065 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:41.503049    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" event={"ID":"27d16f6a-7f81-4142-b920-723e5805a623","Type":"ContainerDied","Data":"1092781c22526f598cec1af475601c77a01da143fa5b44c900760c5b37432134"}
Sep 16 18:59:44.147444 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:44.147368    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7"
Sep 16 18:59:45.057210 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:45.057163    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 16 18:59:46.529074 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:46.529043    2554 generic.go:356] "Generic (PLEG): container finished" podID="27d16f6a-7f81-4142-b920-723e5805a623" containerID="1da8a30e5b9e972a1c531de558c20d3de9f5de13972d2806f5988aff7cbd3211" exitCode=0
Sep 16 18:59:46.529458 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:46.529087    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" event={"ID":"27d16f6a-7f81-4142-b920-723e5805a623","Type":"ContainerDied","Data":"1da8a30e5b9e972a1c531de558c20d3de9f5de13972d2806f5988aff7cbd3211"}
Sep 16 18:59:50.057172 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:50.057127    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 16 18:59:50.062531 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:50.062500    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Sep 16 18:59:50.062910 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:50.062885    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 18:59:50.366731 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:50.366623    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/2.log"
Sep 16 18:59:50.366970 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:50.366953    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/2.log"
Sep 16 18:59:50.372797 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:50.372777    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovn-acl-logging/0.log"
Sep 16 18:59:50.373386 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:50.373365    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovn-acl-logging/0.log"
Sep 16 18:59:51.486347 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:51.486311    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7"]
Sep 16 18:59:51.486836 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:51.486578    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7" podUID="ec4a1895-4ab7-4302-bbf5-4b90a0779228" containerName="kserve-container" containerID="cri-o://8d895b69fd22ebea7906b56915c42e4e62e994111c611428d0fe055e856f2e21" gracePeriod=30
Sep 16 18:59:51.486836 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:51.486606    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7" podUID="ec4a1895-4ab7-4302-bbf5-4b90a0779228" containerName="kube-rbac-proxy" containerID="cri-o://ea317eba517642a4b1f724b1a4117962611a6e266ca13c3f1780c4ccd096a0d6" gracePeriod=30
Sep 16 18:59:52.561055 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:52.561017    2554 generic.go:356] "Generic (PLEG): container finished" podID="ec4a1895-4ab7-4302-bbf5-4b90a0779228" containerID="ea317eba517642a4b1f724b1a4117962611a6e266ca13c3f1780c4ccd096a0d6" exitCode=2
Sep 16 18:59:52.561437 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:52.561099    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7" event={"ID":"ec4a1895-4ab7-4302-bbf5-4b90a0779228","Type":"ContainerDied","Data":"ea317eba517642a4b1f724b1a4117962611a6e266ca13c3f1780c4ccd096a0d6"}
Sep 16 18:59:54.141287 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:54.141247    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7" podUID="ec4a1895-4ab7-4302-bbf5-4b90a0779228" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Sep 16 18:59:54.145998 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:54.145971    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7" podUID="ec4a1895-4ab7-4302-bbf5-4b90a0779228" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Sep 16 18:59:55.057491 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:55.057439    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 16 18:59:55.057674 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:55.057593    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"
Sep 16 18:59:56.329490 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.329462    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7"
Sep 16 18:59:56.352609 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.352579    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec4a1895-4ab7-4302-bbf5-4b90a0779228-kube-rbac-proxy-sar-config\") pod \"ec4a1895-4ab7-4302-bbf5-4b90a0779228\" (UID: \"ec4a1895-4ab7-4302-bbf5-4b90a0779228\") "
Sep 16 18:59:56.352811 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.352674    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec4a1895-4ab7-4302-bbf5-4b90a0779228-kserve-provision-location\") pod \"ec4a1895-4ab7-4302-bbf5-4b90a0779228\" (UID: \"ec4a1895-4ab7-4302-bbf5-4b90a0779228\") "
Sep 16 18:59:56.352811 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.352709    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec4a1895-4ab7-4302-bbf5-4b90a0779228-cabundle-cert\") pod \"ec4a1895-4ab7-4302-bbf5-4b90a0779228\" (UID: \"ec4a1895-4ab7-4302-bbf5-4b90a0779228\") "
Sep 16 18:59:56.352811 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.352777    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec4a1895-4ab7-4302-bbf5-4b90a0779228-proxy-tls\") pod \"ec4a1895-4ab7-4302-bbf5-4b90a0779228\" (UID: \"ec4a1895-4ab7-4302-bbf5-4b90a0779228\") "
Sep 16 18:59:56.352811 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.352802    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvmgm\" (UniqueName: \"kubernetes.io/projected/ec4a1895-4ab7-4302-bbf5-4b90a0779228-kube-api-access-wvmgm\") pod \"ec4a1895-4ab7-4302-bbf5-4b90a0779228\" (UID: \"ec4a1895-4ab7-4302-bbf5-4b90a0779228\") "
Sep 16 18:59:56.353045 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.352956    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4a1895-4ab7-4302-bbf5-4b90a0779228-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ec4a1895-4ab7-4302-bbf5-4b90a0779228" (UID: "ec4a1895-4ab7-4302-bbf5-4b90a0779228"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 18:59:56.353107 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.353063    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec4a1895-4ab7-4302-bbf5-4b90a0779228-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:59:56.353160 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.352956    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec4a1895-4ab7-4302-bbf5-4b90a0779228-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ec4a1895-4ab7-4302-bbf5-4b90a0779228" (UID: "ec4a1895-4ab7-4302-bbf5-4b90a0779228"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:59:56.353314 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.353287    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec4a1895-4ab7-4302-bbf5-4b90a0779228-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ec4a1895-4ab7-4302-bbf5-4b90a0779228" (UID: "ec4a1895-4ab7-4302-bbf5-4b90a0779228"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 18:59:56.355160 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.355121    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec4a1895-4ab7-4302-bbf5-4b90a0779228-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ec4a1895-4ab7-4302-bbf5-4b90a0779228" (UID: "ec4a1895-4ab7-4302-bbf5-4b90a0779228"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 18:59:56.355285 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.355175    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4a1895-4ab7-4302-bbf5-4b90a0779228-kube-api-access-wvmgm" (OuterVolumeSpecName: "kube-api-access-wvmgm") pod "ec4a1895-4ab7-4302-bbf5-4b90a0779228" (UID: "ec4a1895-4ab7-4302-bbf5-4b90a0779228"). InnerVolumeSpecName "kube-api-access-wvmgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 18:59:56.453978 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.453946    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec4a1895-4ab7-4302-bbf5-4b90a0779228-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:59:56.453978 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.453978    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ec4a1895-4ab7-4302-bbf5-4b90a0779228-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:59:56.454201 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.453996    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec4a1895-4ab7-4302-bbf5-4b90a0779228-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:59:56.454201 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.454010    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvmgm\" (UniqueName: \"kubernetes.io/projected/ec4a1895-4ab7-4302-bbf5-4b90a0779228-kube-api-access-wvmgm\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 18:59:56.582494 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.582461    2554 generic.go:356] "Generic (PLEG): container finished" podID="ec4a1895-4ab7-4302-bbf5-4b90a0779228" containerID="8d895b69fd22ebea7906b56915c42e4e62e994111c611428d0fe055e856f2e21" exitCode=0
Sep 16 18:59:56.582676 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.582522    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7"
Sep 16 18:59:56.582676 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.582543    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7" event={"ID":"ec4a1895-4ab7-4302-bbf5-4b90a0779228","Type":"ContainerDied","Data":"8d895b69fd22ebea7906b56915c42e4e62e994111c611428d0fe055e856f2e21"}
Sep 16 18:59:56.582676 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.582621    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7" event={"ID":"ec4a1895-4ab7-4302-bbf5-4b90a0779228","Type":"ContainerDied","Data":"b19692c56d647e95e0518fc0bd46a36153d2f0402ebf5ab8e5dfa89630c0eee3"}
Sep 16 18:59:56.582676 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.582657    2554 scope.go:117] "RemoveContainer" containerID="ea317eba517642a4b1f724b1a4117962611a6e266ca13c3f1780c4ccd096a0d6"
Sep 16 18:59:56.595836 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.593969    2554 scope.go:117] "RemoveContainer" containerID="8d895b69fd22ebea7906b56915c42e4e62e994111c611428d0fe055e856f2e21"
Sep 16 18:59:56.603996 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.603948    2554 scope.go:117] "RemoveContainer" containerID="34de31f5e2928d3b3116fda6b640cf4fcc939a8d7a32a3df23acc0c567fd8aae"
Sep 16 18:59:56.607651 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.607613    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7"]
Sep 16 18:59:56.609873 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.609849    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wz5l7"]
Sep 16 18:59:56.614272 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.614242    2554 scope.go:117] "RemoveContainer" containerID="ea317eba517642a4b1f724b1a4117962611a6e266ca13c3f1780c4ccd096a0d6"
Sep 16 18:59:56.614617 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:59:56.614594    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea317eba517642a4b1f724b1a4117962611a6e266ca13c3f1780c4ccd096a0d6\": container with ID starting with ea317eba517642a4b1f724b1a4117962611a6e266ca13c3f1780c4ccd096a0d6 not found: ID does not exist" containerID="ea317eba517642a4b1f724b1a4117962611a6e266ca13c3f1780c4ccd096a0d6"
Sep 16 18:59:56.614738 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.614627    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea317eba517642a4b1f724b1a4117962611a6e266ca13c3f1780c4ccd096a0d6"} err="failed to get container status \"ea317eba517642a4b1f724b1a4117962611a6e266ca13c3f1780c4ccd096a0d6\": rpc error: code = NotFound desc = could not find container \"ea317eba517642a4b1f724b1a4117962611a6e266ca13c3f1780c4ccd096a0d6\": container with ID starting with ea317eba517642a4b1f724b1a4117962611a6e266ca13c3f1780c4ccd096a0d6 not found: ID does not exist"
Sep 16 18:59:56.614738 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.614647    2554 scope.go:117] "RemoveContainer" containerID="8d895b69fd22ebea7906b56915c42e4e62e994111c611428d0fe055e856f2e21"
Sep 16 18:59:56.614979 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:59:56.614959    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d895b69fd22ebea7906b56915c42e4e62e994111c611428d0fe055e856f2e21\": container with ID starting with 8d895b69fd22ebea7906b56915c42e4e62e994111c611428d0fe055e856f2e21 not found: ID does not exist" containerID="8d895b69fd22ebea7906b56915c42e4e62e994111c611428d0fe055e856f2e21"
Sep 16 18:59:56.615033 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.614983    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d895b69fd22ebea7906b56915c42e4e62e994111c611428d0fe055e856f2e21"} err="failed to get container status \"8d895b69fd22ebea7906b56915c42e4e62e994111c611428d0fe055e856f2e21\": rpc error: code = NotFound desc = could not find container \"8d895b69fd22ebea7906b56915c42e4e62e994111c611428d0fe055e856f2e21\": container with ID starting with 8d895b69fd22ebea7906b56915c42e4e62e994111c611428d0fe055e856f2e21 not found: ID does not exist"
Sep 16 18:59:56.615033 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.614995    2554 scope.go:117] "RemoveContainer" containerID="34de31f5e2928d3b3116fda6b640cf4fcc939a8d7a32a3df23acc0c567fd8aae"
Sep 16 18:59:56.615229 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:59:56.615215    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34de31f5e2928d3b3116fda6b640cf4fcc939a8d7a32a3df23acc0c567fd8aae\": container with ID starting with 34de31f5e2928d3b3116fda6b640cf4fcc939a8d7a32a3df23acc0c567fd8aae not found: ID does not exist" containerID="34de31f5e2928d3b3116fda6b640cf4fcc939a8d7a32a3df23acc0c567fd8aae"
Sep 16 18:59:56.615278 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.615232    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34de31f5e2928d3b3116fda6b640cf4fcc939a8d7a32a3df23acc0c567fd8aae"} err="failed to get container status \"34de31f5e2928d3b3116fda6b640cf4fcc939a8d7a32a3df23acc0c567fd8aae\": rpc error: code = NotFound desc = could not find container \"34de31f5e2928d3b3116fda6b640cf4fcc939a8d7a32a3df23acc0c567fd8aae\": container with ID starting with 34de31f5e2928d3b3116fda6b640cf4fcc939a8d7a32a3df23acc0c567fd8aae not found: ID does not exist"
Sep 16 18:59:56.665514 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.665477    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"]
Sep 16 18:59:56.665919 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.665904    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="storage-initializer"
Sep 16 18:59:56.665972 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.665924    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="storage-initializer"
Sep 16 18:59:56.665972 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.665937    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec4a1895-4ab7-4302-bbf5-4b90a0779228" containerName="kube-rbac-proxy"
Sep 16 18:59:56.665972 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.665943    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec4a1895-4ab7-4302-bbf5-4b90a0779228" containerName="kube-rbac-proxy"
Sep 16 18:59:56.665972 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.665957    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec4a1895-4ab7-4302-bbf5-4b90a0779228" containerName="storage-initializer"
Sep 16 18:59:56.665972 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.665962    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec4a1895-4ab7-4302-bbf5-4b90a0779228" containerName="storage-initializer"
Sep 16 18:59:56.666153 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.665979    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="kube-rbac-proxy"
Sep 16 18:59:56.666153 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.665984    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="kube-rbac-proxy"
Sep 16 18:59:56.666153 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.665998    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="agent"
Sep 16 18:59:56.666153 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.666007    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="agent"
Sep 16 18:59:56.666153 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.666012    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec4a1895-4ab7-4302-bbf5-4b90a0779228" containerName="kserve-container"
Sep 16 18:59:56.666153 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.666019    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec4a1895-4ab7-4302-bbf5-4b90a0779228" containerName="kserve-container"
Sep 16 18:59:56.666153 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.666035    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="kserve-container"
Sep 16 18:59:56.666153 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.666042    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="kserve-container"
Sep 16 18:59:56.666153 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.666101    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="kserve-container"
Sep 16 18:59:56.666153 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.666108    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="kube-rbac-proxy"
Sep 16 18:59:56.666153 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.666117    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="63560122-fda1-4051-b4af-2b6bb701688d" containerName="agent"
Sep 16 18:59:56.666153 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.666123    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec4a1895-4ab7-4302-bbf5-4b90a0779228" containerName="kserve-container"
Sep 16 18:59:56.666153 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.666129    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec4a1895-4ab7-4302-bbf5-4b90a0779228" containerName="kube-rbac-proxy"
Sep 16 18:59:56.670858 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.670838    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"
Sep 16 18:59:56.672991 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.672968    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Sep 16 18:59:56.673110 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.673094    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Sep 16 18:59:56.680671 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.680644    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"]
Sep 16 18:59:56.756448 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.756417    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx\" (UID: \"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"
Sep 16 18:59:56.756648 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.756510    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx\" (UID: \"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"
Sep 16 18:59:56.756648 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.756540    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx\" (UID: \"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"
Sep 16 18:59:56.756648 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.756565    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx\" (UID: \"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"
Sep 16 18:59:56.756648 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.756619    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzfml\" (UniqueName: \"kubernetes.io/projected/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-kube-api-access-tzfml\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx\" (UID: \"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"
Sep 16 18:59:56.857364 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.857325    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx\" (UID: \"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"
Sep 16 18:59:56.857566 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.857369    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx\" (UID: \"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"
Sep 16 18:59:56.857566 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.857399    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx\" (UID: \"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"
Sep 16 18:59:56.857566 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.857465    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzfml\" (UniqueName: \"kubernetes.io/projected/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-kube-api-access-tzfml\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx\" (UID: \"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"
Sep 16 18:59:56.857566 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:59:56.857493    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-serving-cert: secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Sep 16 18:59:56.857566 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.857503    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx\" (UID: \"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"
Sep 16 18:59:56.857566 ip-10-0-137-241 kubenswrapper[2554]: E0916 18:59:56.857560    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-proxy-tls podName:6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2 nodeName:}" failed. No retries permitted until 2026-09-16 18:59:57.357539141 +0000 UTC m=+907.433308758 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-proxy-tls") pod "isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx" (UID: "6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2") : secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Sep 16 18:59:56.858117 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.857916    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx\" (UID: \"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"
Sep 16 18:59:56.858117 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.858107    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx\" (UID: \"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"
Sep 16 18:59:56.858229 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.858200    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx\" (UID: \"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"
Sep 16 18:59:56.868045 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:56.868017    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzfml\" (UniqueName: \"kubernetes.io/projected/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-kube-api-access-tzfml\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx\" (UID: \"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"
Sep 16 18:59:57.360856 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:57.360820    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx\" (UID: \"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"
Sep 16 18:59:57.363274 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:57.363254    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx\" (UID: \"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"
Sep 16 18:59:57.581675 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:57.581627    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"
Sep 16 18:59:57.855495 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:57.855469    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"]
Sep 16 18:59:57.857818 ip-10-0-137-241 kubenswrapper[2554]: W0916 18:59:57.857789    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f8b24ef_03fc_40dc_96bb_d9e9b583b0a2.slice/crio-a0fae8079ebd91e5478c0967f574fdd9efc6048978ae920ff7af0fa7b086aa56 WatchSource:0}: Error finding container a0fae8079ebd91e5478c0967f574fdd9efc6048978ae920ff7af0fa7b086aa56: Status 404 returned error can't find the container with id a0fae8079ebd91e5478c0967f574fdd9efc6048978ae920ff7af0fa7b086aa56
Sep 16 18:59:58.443938 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:58.443905    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec4a1895-4ab7-4302-bbf5-4b90a0779228" path="/var/lib/kubelet/pods/ec4a1895-4ab7-4302-bbf5-4b90a0779228/volumes"
Sep 16 18:59:58.594616 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:58.594580    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx" event={"ID":"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2","Type":"ContainerStarted","Data":"5d3895e762fba39bf3ff761ab940b8bf1804a40e9c243ebeb757c58aa6fd56e2"}
Sep 16 18:59:58.594832 ip-10-0-137-241 kubenswrapper[2554]: I0916 18:59:58.594647    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx" event={"ID":"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2","Type":"ContainerStarted","Data":"a0fae8079ebd91e5478c0967f574fdd9efc6048978ae920ff7af0fa7b086aa56"}
Sep 16 19:00:00.058175 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:00.058133    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 16 19:00:00.062576 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:00.062537    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Sep 16 19:00:00.062928 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:00.062905    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:01.404431 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:01.404397    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"]
Sep 16 19:00:01.411284 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:01.411255    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"
Sep 16 19:00:01.413389 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:01.413361    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Sep 16 19:00:01.413523 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:01.413410    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Sep 16 19:00:01.421694 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:01.421671    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"]
Sep 16 19:00:01.494365 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:01.494332    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/099632f8-19f3-4373-b42c-c5cd508b58fc-cabundle-cert\") pod \"isvc-logger-predictor-768b5b646f-tshh7\" (UID: \"099632f8-19f3-4373-b42c-c5cd508b58fc\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"
Sep 16 19:00:01.494554 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:01.494374    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlsdr\" (UniqueName: \"kubernetes.io/projected/099632f8-19f3-4373-b42c-c5cd508b58fc-kube-api-access-dlsdr\") pod \"isvc-logger-predictor-768b5b646f-tshh7\" (UID: \"099632f8-19f3-4373-b42c-c5cd508b58fc\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"
Sep 16 19:00:01.494554 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:01.494448    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/099632f8-19f3-4373-b42c-c5cd508b58fc-kserve-provision-location\") pod \"isvc-logger-predictor-768b5b646f-tshh7\" (UID: \"099632f8-19f3-4373-b42c-c5cd508b58fc\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"
Sep 16 19:00:01.494654 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:01.494552    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/099632f8-19f3-4373-b42c-c5cd508b58fc-proxy-tls\") pod \"isvc-logger-predictor-768b5b646f-tshh7\" (UID: \"099632f8-19f3-4373-b42c-c5cd508b58fc\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"
Sep 16 19:00:01.494654 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:01.494598    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/099632f8-19f3-4373-b42c-c5cd508b58fc-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-768b5b646f-tshh7\" (UID: \"099632f8-19f3-4373-b42c-c5cd508b58fc\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"
Sep 16 19:00:01.595731 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:01.595680    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/099632f8-19f3-4373-b42c-c5cd508b58fc-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-768b5b646f-tshh7\" (UID: \"099632f8-19f3-4373-b42c-c5cd508b58fc\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"
Sep 16 19:00:01.595912 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:01.595745    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/099632f8-19f3-4373-b42c-c5cd508b58fc-cabundle-cert\") pod \"isvc-logger-predictor-768b5b646f-tshh7\" (UID: \"099632f8-19f3-4373-b42c-c5cd508b58fc\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"
Sep 16 19:00:01.595912 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:01.595779    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlsdr\" (UniqueName: \"kubernetes.io/projected/099632f8-19f3-4373-b42c-c5cd508b58fc-kube-api-access-dlsdr\") pod \"isvc-logger-predictor-768b5b646f-tshh7\" (UID: \"099632f8-19f3-4373-b42c-c5cd508b58fc\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"
Sep 16 19:00:01.595912 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:01.595820    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/099632f8-19f3-4373-b42c-c5cd508b58fc-kserve-provision-location\") pod \"isvc-logger-predictor-768b5b646f-tshh7\" (UID: \"099632f8-19f3-4373-b42c-c5cd508b58fc\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"
Sep 16 19:00:01.595912 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:01.595868    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/099632f8-19f3-4373-b42c-c5cd508b58fc-proxy-tls\") pod \"isvc-logger-predictor-768b5b646f-tshh7\" (UID: \"099632f8-19f3-4373-b42c-c5cd508b58fc\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"
Sep 16 19:00:01.596162 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:00:01.595971    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Sep 16 19:00:01.596162 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:00:01.596039    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/099632f8-19f3-4373-b42c-c5cd508b58fc-proxy-tls podName:099632f8-19f3-4373-b42c-c5cd508b58fc nodeName:}" failed. No retries permitted until 2026-09-16 19:00:02.096018923 +0000 UTC m=+912.171788535 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/099632f8-19f3-4373-b42c-c5cd508b58fc-proxy-tls") pod "isvc-logger-predictor-768b5b646f-tshh7" (UID: "099632f8-19f3-4373-b42c-c5cd508b58fc") : secret "isvc-logger-predictor-serving-cert" not found
Sep 16 19:00:01.596281 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:01.596257    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/099632f8-19f3-4373-b42c-c5cd508b58fc-kserve-provision-location\") pod \"isvc-logger-predictor-768b5b646f-tshh7\" (UID: \"099632f8-19f3-4373-b42c-c5cd508b58fc\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"
Sep 16 19:00:01.596479 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:01.596455    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/099632f8-19f3-4373-b42c-c5cd508b58fc-cabundle-cert\") pod \"isvc-logger-predictor-768b5b646f-tshh7\" (UID: \"099632f8-19f3-4373-b42c-c5cd508b58fc\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"
Sep 16 19:00:01.596747 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:01.596711    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/099632f8-19f3-4373-b42c-c5cd508b58fc-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-768b5b646f-tshh7\" (UID: \"099632f8-19f3-4373-b42c-c5cd508b58fc\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"
Sep 16 19:00:01.604482 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:01.604456    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlsdr\" (UniqueName: \"kubernetes.io/projected/099632f8-19f3-4373-b42c-c5cd508b58fc-kube-api-access-dlsdr\") pod \"isvc-logger-predictor-768b5b646f-tshh7\" (UID: \"099632f8-19f3-4373-b42c-c5cd508b58fc\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"
Sep 16 19:00:02.099896 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:02.099856    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/099632f8-19f3-4373-b42c-c5cd508b58fc-proxy-tls\") pod \"isvc-logger-predictor-768b5b646f-tshh7\" (UID: \"099632f8-19f3-4373-b42c-c5cd508b58fc\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"
Sep 16 19:00:02.102329 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:02.102297    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/099632f8-19f3-4373-b42c-c5cd508b58fc-proxy-tls\") pod \"isvc-logger-predictor-768b5b646f-tshh7\" (UID: \"099632f8-19f3-4373-b42c-c5cd508b58fc\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"
Sep 16 19:00:02.322842 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:02.322787    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"
Sep 16 19:00:02.597076 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:02.597042    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"]
Sep 16 19:00:02.600084 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:00:02.600060    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod099632f8_19f3_4373_b42c_c5cd508b58fc.slice/crio-e90c069c7c96010973981921de491056f5ab4c1c4d5f0532bcc552b489b8dba5 WatchSource:0}: Error finding container e90c069c7c96010973981921de491056f5ab4c1c4d5f0532bcc552b489b8dba5: Status 404 returned error can't find the container with id e90c069c7c96010973981921de491056f5ab4c1c4d5f0532bcc552b489b8dba5
Sep 16 19:00:02.613891 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:02.613861    2554 generic.go:356] "Generic (PLEG): container finished" podID="6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" containerID="5d3895e762fba39bf3ff761ab940b8bf1804a40e9c243ebeb757c58aa6fd56e2" exitCode=0
Sep 16 19:00:02.614010 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:02.613935    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx" event={"ID":"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2","Type":"ContainerDied","Data":"5d3895e762fba39bf3ff761ab940b8bf1804a40e9c243ebeb757c58aa6fd56e2"}
Sep 16 19:00:02.615600 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:02.615573    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" event={"ID":"099632f8-19f3-4373-b42c-c5cd508b58fc","Type":"ContainerStarted","Data":"e90c069c7c96010973981921de491056f5ab4c1c4d5f0532bcc552b489b8dba5"}
Sep 16 19:00:03.622378 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:03.622332    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx" event={"ID":"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2","Type":"ContainerStarted","Data":"18d5bf02c4e96c419caa75800330cd26c73ce85c3b94f489229066128720d4e2"}
Sep 16 19:00:03.622833 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:03.622427    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx" event={"ID":"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2","Type":"ContainerStarted","Data":"8eea51fcafccd5aa3ba848515fd80640d55593b5e70bcc93781390b7db5adb34"}
Sep 16 19:00:03.622833 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:03.622744    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"
Sep 16 19:00:03.622965 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:03.622889    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"
Sep 16 19:00:03.624156 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:03.624127    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" event={"ID":"099632f8-19f3-4373-b42c-c5cd508b58fc","Type":"ContainerStarted","Data":"770d9cb7a5ba8e1084037dfc2900dd6d3d70aafece918d761e3c507783fdb6a4"}
Sep 16 19:00:03.624460 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:03.624434    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx" podUID="6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:00:03.642192 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:03.641941    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx" podStartSLOduration=3.9197598129999998 podStartE2EDuration="7.641923617s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.722163804s" isStatefulPod=false podCreationTimestamp="2026-09-16 18:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:03.641346286 +0000 UTC m=+913.717115918" watchObservedRunningTime="2026-09-16 19:00:03.641923617 +0000 UTC m=+913.717693280"
Sep 16 19:00:04.629252 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:04.629214    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx" podUID="6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:00:05.057262 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:05.057216    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 16 19:00:06.641044 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:06.641012    2554 generic.go:356] "Generic (PLEG): container finished" podID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerID="770d9cb7a5ba8e1084037dfc2900dd6d3d70aafece918d761e3c507783fdb6a4" exitCode=0
Sep 16 19:00:06.641436 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:06.641091    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" event={"ID":"099632f8-19f3-4373-b42c-c5cd508b58fc","Type":"ContainerDied","Data":"770d9cb7a5ba8e1084037dfc2900dd6d3d70aafece918d761e3c507783fdb6a4"}
Sep 16 19:00:07.648525 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:07.648492    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" event={"ID":"099632f8-19f3-4373-b42c-c5cd508b58fc","Type":"ContainerStarted","Data":"53e1e563bfddd449ae6107f4a58fb5f581112c65255e4db23e330add1777cefb"}
Sep 16 19:00:07.648949 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:07.648559    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" event={"ID":"099632f8-19f3-4373-b42c-c5cd508b58fc","Type":"ContainerStarted","Data":"1796578f51addde3f088be39a3cff7a8af4ed468e591e21b4190338dcd4aad61"}
Sep 16 19:00:07.648949 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:07.648582    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" event={"ID":"099632f8-19f3-4373-b42c-c5cd508b58fc","Type":"ContainerStarted","Data":"1b4fabf3e8a4ee7e6bd8b5d2ab072c9390e3abb9402108d30e3d077c9267f907"}
Sep 16 19:00:07.648949 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:07.648874    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"
Sep 16 19:00:07.648949 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:07.648937    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"
Sep 16 19:00:07.649102 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:07.648969    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"
Sep 16 19:00:07.650605 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:07.650574    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:00:07.651339 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:07.651307    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:07.673880 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:07.673839    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podStartSLOduration=2.849369179 podStartE2EDuration="6.673827239s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.82445806s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:00:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:00:07.671291424 +0000 UTC m=+917.747061057" watchObservedRunningTime="2026-09-16 19:00:07.673827239 +0000 UTC m=+917.749596870"
Sep 16 19:00:08.654210 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:08.654163    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:00:08.654776 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:08.654750    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:09.633555 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:09.633528    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"
Sep 16 19:00:09.634037 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:09.634011    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx" podUID="6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:00:10.058188 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:10.058140    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Sep 16 19:00:10.062477 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:10.062452    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:5000: connect: connection refused"
Sep 16 19:00:10.062573 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:10.062558    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"
Sep 16 19:00:10.062881 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:10.062857    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:10.062995 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:10.062982    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"
Sep 16 19:00:11.403289 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.403266    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"
Sep 16 19:00:11.482450 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.482402    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27d16f6a-7f81-4142-b920-723e5805a623-cabundle-cert\") pod \"27d16f6a-7f81-4142-b920-723e5805a623\" (UID: \"27d16f6a-7f81-4142-b920-723e5805a623\") "
Sep 16 19:00:11.482450 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.482450    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27d16f6a-7f81-4142-b920-723e5805a623-kserve-provision-location\") pod \"27d16f6a-7f81-4142-b920-723e5805a623\" (UID: \"27d16f6a-7f81-4142-b920-723e5805a623\") "
Sep 16 19:00:11.482667 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.482521    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9s7nd\" (UniqueName: \"kubernetes.io/projected/27d16f6a-7f81-4142-b920-723e5805a623-kube-api-access-9s7nd\") pod \"27d16f6a-7f81-4142-b920-723e5805a623\" (UID: \"27d16f6a-7f81-4142-b920-723e5805a623\") "
Sep 16 19:00:11.482667 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.482564    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27d16f6a-7f81-4142-b920-723e5805a623-proxy-tls\") pod \"27d16f6a-7f81-4142-b920-723e5805a623\" (UID: \"27d16f6a-7f81-4142-b920-723e5805a623\") "
Sep 16 19:00:11.482667 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.482581    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27d16f6a-7f81-4142-b920-723e5805a623-kube-rbac-proxy-sar-config\") pod \"27d16f6a-7f81-4142-b920-723e5805a623\" (UID: \"27d16f6a-7f81-4142-b920-723e5805a623\") "
Sep 16 19:00:11.482826 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.482772    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27d16f6a-7f81-4142-b920-723e5805a623-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "27d16f6a-7f81-4142-b920-723e5805a623" (UID: "27d16f6a-7f81-4142-b920-723e5805a623"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:00:11.482826 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.482803    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d16f6a-7f81-4142-b920-723e5805a623-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "27d16f6a-7f81-4142-b920-723e5805a623" (UID: "27d16f6a-7f81-4142-b920-723e5805a623"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:11.483089 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.483063    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d16f6a-7f81-4142-b920-723e5805a623-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "27d16f6a-7f81-4142-b920-723e5805a623" (UID: "27d16f6a-7f81-4142-b920-723e5805a623"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:00:11.484643 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.484624    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d16f6a-7f81-4142-b920-723e5805a623-kube-api-access-9s7nd" (OuterVolumeSpecName: "kube-api-access-9s7nd") pod "27d16f6a-7f81-4142-b920-723e5805a623" (UID: "27d16f6a-7f81-4142-b920-723e5805a623"). InnerVolumeSpecName "kube-api-access-9s7nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:00:11.484707 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.484623    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27d16f6a-7f81-4142-b920-723e5805a623-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "27d16f6a-7f81-4142-b920-723e5805a623" (UID: "27d16f6a-7f81-4142-b920-723e5805a623"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:00:11.583211 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.583148    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/27d16f6a-7f81-4142-b920-723e5805a623-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:00:11.583211 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.583169    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/27d16f6a-7f81-4142-b920-723e5805a623-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:00:11.583211 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.583179    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s7nd\" (UniqueName: \"kubernetes.io/projected/27d16f6a-7f81-4142-b920-723e5805a623-kube-api-access-9s7nd\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:00:11.583211 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.583188    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/27d16f6a-7f81-4142-b920-723e5805a623-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:00:11.583211 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.583196    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/27d16f6a-7f81-4142-b920-723e5805a623-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:00:11.672553 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.672524    2554 generic.go:356] "Generic (PLEG): container finished" podID="27d16f6a-7f81-4142-b920-723e5805a623" containerID="9a96dd75d7d7ace482d072d4b4d2d71df085e23bbc726f1e65d8b50f41086b05" exitCode=0
Sep 16 19:00:11.672695 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.672568    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" event={"ID":"27d16f6a-7f81-4142-b920-723e5805a623","Type":"ContainerDied","Data":"9a96dd75d7d7ace482d072d4b4d2d71df085e23bbc726f1e65d8b50f41086b05"}
Sep 16 19:00:11.672695 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.672578    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"
Sep 16 19:00:11.672695 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.672626    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql" event={"ID":"27d16f6a-7f81-4142-b920-723e5805a623","Type":"ContainerDied","Data":"763fb517177bfc0b16d8aef63a0420da8d4d6dba7390553d6bfff07f3141e134"}
Sep 16 19:00:11.672695 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.672659    2554 scope.go:117] "RemoveContainer" containerID="9a96dd75d7d7ace482d072d4b4d2d71df085e23bbc726f1e65d8b50f41086b05"
Sep 16 19:00:11.681264 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.681244    2554 scope.go:117] "RemoveContainer" containerID="1092781c22526f598cec1af475601c77a01da143fa5b44c900760c5b37432134"
Sep 16 19:00:11.688546 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.688529    2554 scope.go:117] "RemoveContainer" containerID="1da8a30e5b9e972a1c531de558c20d3de9f5de13972d2806f5988aff7cbd3211"
Sep 16 19:00:11.698124 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.695477    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"]
Sep 16 19:00:11.698124 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.696522    2554 scope.go:117] "RemoveContainer" containerID="86b9992d1428ad4ad86ace0fd4722c680fc368f3ac51c8a0dac2d2335fa05bc1"
Sep 16 19:00:11.701430 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.701408    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-f8576c5d4-r7wql"]
Sep 16 19:00:11.705485 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.705471    2554 scope.go:117] "RemoveContainer" containerID="9a96dd75d7d7ace482d072d4b4d2d71df085e23bbc726f1e65d8b50f41086b05"
Sep 16 19:00:11.705778 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:00:11.705760    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a96dd75d7d7ace482d072d4b4d2d71df085e23bbc726f1e65d8b50f41086b05\": container with ID starting with 9a96dd75d7d7ace482d072d4b4d2d71df085e23bbc726f1e65d8b50f41086b05 not found: ID does not exist" containerID="9a96dd75d7d7ace482d072d4b4d2d71df085e23bbc726f1e65d8b50f41086b05"
Sep 16 19:00:11.705835 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.705787    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a96dd75d7d7ace482d072d4b4d2d71df085e23bbc726f1e65d8b50f41086b05"} err="failed to get container status \"9a96dd75d7d7ace482d072d4b4d2d71df085e23bbc726f1e65d8b50f41086b05\": rpc error: code = NotFound desc = could not find container \"9a96dd75d7d7ace482d072d4b4d2d71df085e23bbc726f1e65d8b50f41086b05\": container with ID starting with 9a96dd75d7d7ace482d072d4b4d2d71df085e23bbc726f1e65d8b50f41086b05 not found: ID does not exist"
Sep 16 19:00:11.705835 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.705803    2554 scope.go:117] "RemoveContainer" containerID="1092781c22526f598cec1af475601c77a01da143fa5b44c900760c5b37432134"
Sep 16 19:00:11.706053 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:00:11.706033    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1092781c22526f598cec1af475601c77a01da143fa5b44c900760c5b37432134\": container with ID starting with 1092781c22526f598cec1af475601c77a01da143fa5b44c900760c5b37432134 not found: ID does not exist" containerID="1092781c22526f598cec1af475601c77a01da143fa5b44c900760c5b37432134"
Sep 16 19:00:11.706094 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.706061    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1092781c22526f598cec1af475601c77a01da143fa5b44c900760c5b37432134"} err="failed to get container status \"1092781c22526f598cec1af475601c77a01da143fa5b44c900760c5b37432134\": rpc error: code = NotFound desc = could not find container \"1092781c22526f598cec1af475601c77a01da143fa5b44c900760c5b37432134\": container with ID starting with 1092781c22526f598cec1af475601c77a01da143fa5b44c900760c5b37432134 not found: ID does not exist"
Sep 16 19:00:11.706094 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.706082    2554 scope.go:117] "RemoveContainer" containerID="1da8a30e5b9e972a1c531de558c20d3de9f5de13972d2806f5988aff7cbd3211"
Sep 16 19:00:11.706315 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:00:11.706297    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1da8a30e5b9e972a1c531de558c20d3de9f5de13972d2806f5988aff7cbd3211\": container with ID starting with 1da8a30e5b9e972a1c531de558c20d3de9f5de13972d2806f5988aff7cbd3211 not found: ID does not exist" containerID="1da8a30e5b9e972a1c531de558c20d3de9f5de13972d2806f5988aff7cbd3211"
Sep 16 19:00:11.706353 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.706321    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1da8a30e5b9e972a1c531de558c20d3de9f5de13972d2806f5988aff7cbd3211"} err="failed to get container status \"1da8a30e5b9e972a1c531de558c20d3de9f5de13972d2806f5988aff7cbd3211\": rpc error: code = NotFound desc = could not find container \"1da8a30e5b9e972a1c531de558c20d3de9f5de13972d2806f5988aff7cbd3211\": container with ID starting with 1da8a30e5b9e972a1c531de558c20d3de9f5de13972d2806f5988aff7cbd3211 not found: ID does not exist"
Sep 16 19:00:11.706353 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.706336    2554 scope.go:117] "RemoveContainer" containerID="86b9992d1428ad4ad86ace0fd4722c680fc368f3ac51c8a0dac2d2335fa05bc1"
Sep 16 19:00:11.706554 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:00:11.706538    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86b9992d1428ad4ad86ace0fd4722c680fc368f3ac51c8a0dac2d2335fa05bc1\": container with ID starting with 86b9992d1428ad4ad86ace0fd4722c680fc368f3ac51c8a0dac2d2335fa05bc1 not found: ID does not exist" containerID="86b9992d1428ad4ad86ace0fd4722c680fc368f3ac51c8a0dac2d2335fa05bc1"
Sep 16 19:00:11.706590 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:11.706558    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86b9992d1428ad4ad86ace0fd4722c680fc368f3ac51c8a0dac2d2335fa05bc1"} err="failed to get container status \"86b9992d1428ad4ad86ace0fd4722c680fc368f3ac51c8a0dac2d2335fa05bc1\": rpc error: code = NotFound desc = could not find container \"86b9992d1428ad4ad86ace0fd4722c680fc368f3ac51c8a0dac2d2335fa05bc1\": container with ID starting with 86b9992d1428ad4ad86ace0fd4722c680fc368f3ac51c8a0dac2d2335fa05bc1 not found: ID does not exist"
Sep 16 19:00:12.443496 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:12.443467    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27d16f6a-7f81-4142-b920-723e5805a623" path="/var/lib/kubelet/pods/27d16f6a-7f81-4142-b920-723e5805a623/volumes"
Sep 16 19:00:13.659669 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:13.659637    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"
Sep 16 19:00:13.660292 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:13.660258    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:00:13.660616 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:13.660591    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:19.634418 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:19.634377    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx" podUID="6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:00:23.660929 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:23.660886    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:00:23.661445 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:23.661378    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:29.634344 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:29.634301    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx" podUID="6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:00:33.660376 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:33.660328    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:00:33.663015 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:33.660868    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:39.634635 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:39.634596    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx" podUID="6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:00:43.661153 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:43.661111    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:00:43.661625 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:43.661534    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:49.634200 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:49.634154    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx" podUID="6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:00:53.660665 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:53.660624    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:00:53.661157 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:53.661130    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:00:59.634638 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:00:59.634595    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx" podUID="6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:01:03.660615 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:03.660567    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:01:03.661059 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:03.661038    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:01:09.634223 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:09.634166    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx" podUID="6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:01:13.660641 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:13.660529    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:01:13.661109 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:13.661084    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:01:19.634713 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:19.634678    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"
Sep 16 19:01:23.660923 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:23.660894    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"
Sep 16 19:01:23.661548 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:23.661527    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"
Sep 16 19:01:26.765625 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:26.765589    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"]
Sep 16 19:01:26.766154 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:26.765836    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx" podUID="6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" containerName="kserve-container" containerID="cri-o://8eea51fcafccd5aa3ba848515fd80640d55593b5e70bcc93781390b7db5adb34" gracePeriod=30
Sep 16 19:01:26.766154 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:26.765927    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx" podUID="6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" containerName="kube-rbac-proxy" containerID="cri-o://18d5bf02c4e96c419caa75800330cd26c73ce85c3b94f489229066128720d4e2" gracePeriod=30
Sep 16 19:01:27.042376 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:27.042282    2554 generic.go:356] "Generic (PLEG): container finished" podID="6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" containerID="18d5bf02c4e96c419caa75800330cd26c73ce85c3b94f489229066128720d4e2" exitCode=2
Sep 16 19:01:27.042376 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:27.042347    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx" event={"ID":"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2","Type":"ContainerDied","Data":"18d5bf02c4e96c419caa75800330cd26c73ce85c3b94f489229066128720d4e2"}
Sep 16 19:01:29.630210 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:29.630167    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx" podUID="6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Sep 16 19:01:29.634566 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:29.634541    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx" podUID="6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Sep 16 19:01:31.808987 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:31.808957    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"
Sep 16 19:01:31.998977 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:31.998944    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-kserve-provision-location\") pod \"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2\" (UID: \"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2\") "
Sep 16 19:01:31.999180 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:31.999028    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-kube-rbac-proxy-sar-config\") pod \"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2\" (UID: \"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2\") "
Sep 16 19:01:31.999180 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:31.999057    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-proxy-tls\") pod \"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2\" (UID: \"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2\") "
Sep 16 19:01:31.999180 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:31.999118    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzfml\" (UniqueName: \"kubernetes.io/projected/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-kube-api-access-tzfml\") pod \"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2\" (UID: \"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2\") "
Sep 16 19:01:31.999359 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:31.999181    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-cabundle-cert\") pod \"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2\" (UID: \"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2\") "
Sep 16 19:01:31.999411 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:31.999355    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" (UID: "6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:01:31.999472 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:31.999450    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" (UID: "6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:31.999602 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:31.999580    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" (UID: "6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:01:31.999647 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:31.999589    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:01:31.999647 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:31.999631    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:01:32.001290 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.001261    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" (UID: "6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:01:32.001548 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.001528    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-kube-api-access-tzfml" (OuterVolumeSpecName: "kube-api-access-tzfml") pod "6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" (UID: "6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2"). InnerVolumeSpecName "kube-api-access-tzfml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:01:32.049129 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.049092    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"]
Sep 16 19:01:32.049466 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.049453    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" containerName="kube-rbac-proxy"
Sep 16 19:01:32.049507 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.049469    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" containerName="kube-rbac-proxy"
Sep 16 19:01:32.049507 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.049485    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" containerName="storage-initializer"
Sep 16 19:01:32.049507 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.049490    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" containerName="storage-initializer"
Sep 16 19:01:32.049507 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.049500    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="kserve-container"
Sep 16 19:01:32.049507 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.049506    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="kserve-container"
Sep 16 19:01:32.049661 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.049514    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="storage-initializer"
Sep 16 19:01:32.049661 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.049519    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="storage-initializer"
Sep 16 19:01:32.049661 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.049526    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="agent"
Sep 16 19:01:32.049661 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.049533    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="agent"
Sep 16 19:01:32.049661 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.049537    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" containerName="kserve-container"
Sep 16 19:01:32.049661 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.049543    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" containerName="kserve-container"
Sep 16 19:01:32.049661 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.049551    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="kube-rbac-proxy"
Sep 16 19:01:32.049661 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.049555    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="kube-rbac-proxy"
Sep 16 19:01:32.049661 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.049619    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" containerName="kube-rbac-proxy"
Sep 16 19:01:32.049661 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.049627    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="agent"
Sep 16 19:01:32.049661 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.049632    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="kube-rbac-proxy"
Sep 16 19:01:32.049661 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.049638    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="27d16f6a-7f81-4142-b920-723e5805a623" containerName="kserve-container"
Sep 16 19:01:32.049661 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.049644    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" containerName="kserve-container"
Sep 16 19:01:32.052986 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.052971    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"
Sep 16 19:01:32.054942 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.054920    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Sep 16 19:01:32.055276 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.055256    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 16 19:01:32.064911 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.064885    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"]
Sep 16 19:01:32.076174 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.076139    2554 generic.go:356] "Generic (PLEG): container finished" podID="6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" containerID="8eea51fcafccd5aa3ba848515fd80640d55593b5e70bcc93781390b7db5adb34" exitCode=0
Sep 16 19:01:32.076347 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.076176    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx" event={"ID":"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2","Type":"ContainerDied","Data":"8eea51fcafccd5aa3ba848515fd80640d55593b5e70bcc93781390b7db5adb34"}
Sep 16 19:01:32.076347 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.076212    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"
Sep 16 19:01:32.076347 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.076255    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx" event={"ID":"6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2","Type":"ContainerDied","Data":"a0fae8079ebd91e5478c0967f574fdd9efc6048978ae920ff7af0fa7b086aa56"}
Sep 16 19:01:32.076347 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.076300    2554 scope.go:117] "RemoveContainer" containerID="18d5bf02c4e96c419caa75800330cd26c73ce85c3b94f489229066128720d4e2"
Sep 16 19:01:32.085831 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.085809    2554 scope.go:117] "RemoveContainer" containerID="8eea51fcafccd5aa3ba848515fd80640d55593b5e70bcc93781390b7db5adb34"
Sep 16 19:01:32.093758 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.093741    2554 scope.go:117] "RemoveContainer" containerID="5d3895e762fba39bf3ff761ab940b8bf1804a40e9c243ebeb757c58aa6fd56e2"
Sep 16 19:01:32.099894 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.099871    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"]
Sep 16 19:01:32.100356 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.100335    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:01:32.100400 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.100364    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:01:32.100400 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.100378    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzfml\" (UniqueName: \"kubernetes.io/projected/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2-kube-api-access-tzfml\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:01:32.101737 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.101703    2554 scope.go:117] "RemoveContainer" containerID="18d5bf02c4e96c419caa75800330cd26c73ce85c3b94f489229066128720d4e2"
Sep 16 19:01:32.102051 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:01:32.102027    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18d5bf02c4e96c419caa75800330cd26c73ce85c3b94f489229066128720d4e2\": container with ID starting with 18d5bf02c4e96c419caa75800330cd26c73ce85c3b94f489229066128720d4e2 not found: ID does not exist" containerID="18d5bf02c4e96c419caa75800330cd26c73ce85c3b94f489229066128720d4e2"
Sep 16 19:01:32.102149 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.102054    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18d5bf02c4e96c419caa75800330cd26c73ce85c3b94f489229066128720d4e2"} err="failed to get container status \"18d5bf02c4e96c419caa75800330cd26c73ce85c3b94f489229066128720d4e2\": rpc error: code = NotFound desc = could not find container \"18d5bf02c4e96c419caa75800330cd26c73ce85c3b94f489229066128720d4e2\": container with ID starting with 18d5bf02c4e96c419caa75800330cd26c73ce85c3b94f489229066128720d4e2 not found: ID does not exist"
Sep 16 19:01:32.102149 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.102072    2554 scope.go:117] "RemoveContainer" containerID="8eea51fcafccd5aa3ba848515fd80640d55593b5e70bcc93781390b7db5adb34"
Sep 16 19:01:32.102539 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:01:32.102511    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eea51fcafccd5aa3ba848515fd80640d55593b5e70bcc93781390b7db5adb34\": container with ID starting with 8eea51fcafccd5aa3ba848515fd80640d55593b5e70bcc93781390b7db5adb34 not found: ID does not exist" containerID="8eea51fcafccd5aa3ba848515fd80640d55593b5e70bcc93781390b7db5adb34"
Sep 16 19:01:32.102665 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.102546    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eea51fcafccd5aa3ba848515fd80640d55593b5e70bcc93781390b7db5adb34"} err="failed to get container status \"8eea51fcafccd5aa3ba848515fd80640d55593b5e70bcc93781390b7db5adb34\": rpc error: code = NotFound desc = could not find container \"8eea51fcafccd5aa3ba848515fd80640d55593b5e70bcc93781390b7db5adb34\": container with ID starting with 8eea51fcafccd5aa3ba848515fd80640d55593b5e70bcc93781390b7db5adb34 not found: ID does not exist"
Sep 16 19:01:32.102665 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.102588    2554 scope.go:117] "RemoveContainer" containerID="5d3895e762fba39bf3ff761ab940b8bf1804a40e9c243ebeb757c58aa6fd56e2"
Sep 16 19:01:32.102975 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:01:32.102941    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d3895e762fba39bf3ff761ab940b8bf1804a40e9c243ebeb757c58aa6fd56e2\": container with ID starting with 5d3895e762fba39bf3ff761ab940b8bf1804a40e9c243ebeb757c58aa6fd56e2 not found: ID does not exist" containerID="5d3895e762fba39bf3ff761ab940b8bf1804a40e9c243ebeb757c58aa6fd56e2"
Sep 16 19:01:32.103060 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.102969    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d3895e762fba39bf3ff761ab940b8bf1804a40e9c243ebeb757c58aa6fd56e2"} err="failed to get container status \"5d3895e762fba39bf3ff761ab940b8bf1804a40e9c243ebeb757c58aa6fd56e2\": rpc error: code = NotFound desc = could not find container \"5d3895e762fba39bf3ff761ab940b8bf1804a40e9c243ebeb757c58aa6fd56e2\": container with ID starting with 5d3895e762fba39bf3ff761ab940b8bf1804a40e9c243ebeb757c58aa6fd56e2 not found: ID does not exist"
Sep 16 19:01:32.104581 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.104562    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-t2rlx"]
Sep 16 19:01:32.201693 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.201649    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eead0240-eac0-44df-b985-821bce775fbb-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z\" (UID: \"eead0240-eac0-44df-b985-821bce775fbb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"
Sep 16 19:01:32.201693 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.201692    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eead0240-eac0-44df-b985-821bce775fbb-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z\" (UID: \"eead0240-eac0-44df-b985-821bce775fbb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"
Sep 16 19:01:32.201930 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.201747    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eead0240-eac0-44df-b985-821bce775fbb-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z\" (UID: \"eead0240-eac0-44df-b985-821bce775fbb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"
Sep 16 19:01:32.201930 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.201801    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eead0240-eac0-44df-b985-821bce775fbb-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z\" (UID: \"eead0240-eac0-44df-b985-821bce775fbb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"
Sep 16 19:01:32.201930 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.201832    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxscp\" (UniqueName: \"kubernetes.io/projected/eead0240-eac0-44df-b985-821bce775fbb-kube-api-access-bxscp\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z\" (UID: \"eead0240-eac0-44df-b985-821bce775fbb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"
Sep 16 19:01:32.303389 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.303286    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eead0240-eac0-44df-b985-821bce775fbb-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z\" (UID: \"eead0240-eac0-44df-b985-821bce775fbb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"
Sep 16 19:01:32.303389 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.303331    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eead0240-eac0-44df-b985-821bce775fbb-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z\" (UID: \"eead0240-eac0-44df-b985-821bce775fbb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"
Sep 16 19:01:32.303389 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.303352    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eead0240-eac0-44df-b985-821bce775fbb-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z\" (UID: \"eead0240-eac0-44df-b985-821bce775fbb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"
Sep 16 19:01:32.303389 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.303379    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eead0240-eac0-44df-b985-821bce775fbb-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z\" (UID: \"eead0240-eac0-44df-b985-821bce775fbb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"
Sep 16 19:01:32.303693 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.303424    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxscp\" (UniqueName: \"kubernetes.io/projected/eead0240-eac0-44df-b985-821bce775fbb-kube-api-access-bxscp\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z\" (UID: \"eead0240-eac0-44df-b985-821bce775fbb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"
Sep 16 19:01:32.303847 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.303822    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eead0240-eac0-44df-b985-821bce775fbb-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z\" (UID: \"eead0240-eac0-44df-b985-821bce775fbb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"
Sep 16 19:01:32.304210 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.304187    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eead0240-eac0-44df-b985-821bce775fbb-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z\" (UID: \"eead0240-eac0-44df-b985-821bce775fbb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"
Sep 16 19:01:32.304263 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.304187    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eead0240-eac0-44df-b985-821bce775fbb-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z\" (UID: \"eead0240-eac0-44df-b985-821bce775fbb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"
Sep 16 19:01:32.305900 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.305879    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eead0240-eac0-44df-b985-821bce775fbb-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z\" (UID: \"eead0240-eac0-44df-b985-821bce775fbb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"
Sep 16 19:01:32.311958 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.311931    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxscp\" (UniqueName: \"kubernetes.io/projected/eead0240-eac0-44df-b985-821bce775fbb-kube-api-access-bxscp\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z\" (UID: \"eead0240-eac0-44df-b985-821bce775fbb\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"
Sep 16 19:01:32.363515 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.363476    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"
Sep 16 19:01:32.444394 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.444363    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2" path="/var/lib/kubelet/pods/6f8b24ef-03fc-40dc-96bb-d9e9b583b0a2/volumes"
Sep 16 19:01:32.641794 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:32.641764    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"]
Sep 16 19:01:32.644533 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:01:32.644509    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeead0240_eac0_44df_b985_821bce775fbb.slice/crio-f620807c713e4d8b5de4b5486a331d87944941c803b2f06f16d307117aca5ffd WatchSource:0}: Error finding container f620807c713e4d8b5de4b5486a331d87944941c803b2f06f16d307117aca5ffd: Status 404 returned error can't find the container with id f620807c713e4d8b5de4b5486a331d87944941c803b2f06f16d307117aca5ffd
Sep 16 19:01:33.083506 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:33.083468    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z" event={"ID":"eead0240-eac0-44df-b985-821bce775fbb","Type":"ContainerStarted","Data":"9eb7f5d49715911c3369c856b4f62fd0b6ecc40086a5f246eb3af8c661365caf"}
Sep 16 19:01:33.083941 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:33.083534    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z" event={"ID":"eead0240-eac0-44df-b985-821bce775fbb","Type":"ContainerStarted","Data":"f620807c713e4d8b5de4b5486a331d87944941c803b2f06f16d307117aca5ffd"}
Sep 16 19:01:36.608885 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:36.608843    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"]
Sep 16 19:01:36.609321 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:36.609195    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="kserve-container" containerID="cri-o://1b4fabf3e8a4ee7e6bd8b5d2ab072c9390e3abb9402108d30e3d077c9267f907" gracePeriod=30
Sep 16 19:01:36.609321 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:36.609289    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="agent" containerID="cri-o://53e1e563bfddd449ae6107f4a58fb5f581112c65255e4db23e330add1777cefb" gracePeriod=30
Sep 16 19:01:36.609440 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:36.609381    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="kube-rbac-proxy" containerID="cri-o://1796578f51addde3f088be39a3cff7a8af4ed468e591e21b4190338dcd4aad61" gracePeriod=30
Sep 16 19:01:36.739372 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:36.739337    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"]
Sep 16 19:01:36.743201 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:36.743177    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"
Sep 16 19:01:36.745267 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:36.745242    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Sep 16 19:01:36.745395 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:36.745301    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Sep 16 19:01:36.754900 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:36.754876    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"]
Sep 16 19:01:36.840173 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:36.840134    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2b5c5\" (UID: \"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"
Sep 16 19:01:36.840173 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:36.840178    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbp65\" (UniqueName: \"kubernetes.io/projected/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-kube-api-access-zbp65\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2b5c5\" (UID: \"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"
Sep 16 19:01:36.840394 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:36.840251    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2b5c5\" (UID: \"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"
Sep 16 19:01:36.840394 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:36.840278    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2b5c5\" (UID: \"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"
Sep 16 19:01:36.840394 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:36.840307    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2b5c5\" (UID: \"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"
Sep 16 19:01:36.941306 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:36.941215    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2b5c5\" (UID: \"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"
Sep 16 19:01:36.941306 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:36.941260    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbp65\" (UniqueName: \"kubernetes.io/projected/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-kube-api-access-zbp65\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2b5c5\" (UID: \"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"
Sep 16 19:01:36.941306 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:36.941296    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2b5c5\" (UID: \"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"
Sep 16 19:01:36.941608 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:36.941323    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2b5c5\" (UID: \"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"
Sep 16 19:01:36.941608 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:36.941356    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2b5c5\" (UID: \"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"
Sep 16 19:01:36.941608 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:01:36.941451    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v1-predictor-serving-cert: secret "isvc-autogluon-v1-predictor-serving-cert" not found
Sep 16 19:01:36.941608 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:01:36.941523    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-proxy-tls podName:e4b77abe-3eb5-41dc-9e3e-f08f8a98537d nodeName:}" failed. No retries permitted until 2026-09-16 19:01:37.441502475 +0000 UTC m=+1007.517272086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-proxy-tls") pod "isvc-autogluon-v1-predictor-574b5b7495-2b5c5" (UID: "e4b77abe-3eb5-41dc-9e3e-f08f8a98537d") : secret "isvc-autogluon-v1-predictor-serving-cert" not found
Sep 16 19:01:36.941869 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:36.941822    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2b5c5\" (UID: \"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"
Sep 16 19:01:36.942027 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:36.942007    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2b5c5\" (UID: \"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"
Sep 16 19:01:36.942063 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:36.942010    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2b5c5\" (UID: \"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"
Sep 16 19:01:36.949986 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:36.949961    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbp65\" (UniqueName: \"kubernetes.io/projected/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-kube-api-access-zbp65\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2b5c5\" (UID: \"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"
Sep 16 19:01:37.106016 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:37.105982    2554 generic.go:356] "Generic (PLEG): container finished" podID="eead0240-eac0-44df-b985-821bce775fbb" containerID="9eb7f5d49715911c3369c856b4f62fd0b6ecc40086a5f246eb3af8c661365caf" exitCode=0
Sep 16 19:01:37.106208 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:37.106057    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z" event={"ID":"eead0240-eac0-44df-b985-821bce775fbb","Type":"ContainerDied","Data":"9eb7f5d49715911c3369c856b4f62fd0b6ecc40086a5f246eb3af8c661365caf"}
Sep 16 19:01:37.108838 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:37.108799    2554 generic.go:356] "Generic (PLEG): container finished" podID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerID="1796578f51addde3f088be39a3cff7a8af4ed468e591e21b4190338dcd4aad61" exitCode=2
Sep 16 19:01:37.108963 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:37.108860    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" event={"ID":"099632f8-19f3-4373-b42c-c5cd508b58fc","Type":"ContainerDied","Data":"1796578f51addde3f088be39a3cff7a8af4ed468e591e21b4190338dcd4aad61"}
Sep 16 19:01:37.446676 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:37.446634    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2b5c5\" (UID: \"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"
Sep 16 19:01:37.449095 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:37.449075    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-proxy-tls\") pod \"isvc-autogluon-v1-predictor-574b5b7495-2b5c5\" (UID: \"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"
Sep 16 19:01:37.653839 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:37.653799    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"
Sep 16 19:01:37.933755 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:37.933704    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"]
Sep 16 19:01:37.934978 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:01:37.934942    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4b77abe_3eb5_41dc_9e3e_f08f8a98537d.slice/crio-bd010e4b8e0d8490c4d518eb634c48f42ad4673ea62768995f535c94c86d291b WatchSource:0}: Error finding container bd010e4b8e0d8490c4d518eb634c48f42ad4673ea62768995f535c94c86d291b: Status 404 returned error can't find the container with id bd010e4b8e0d8490c4d518eb634c48f42ad4673ea62768995f535c94c86d291b
Sep 16 19:01:38.116113 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:38.116080    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5" event={"ID":"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d","Type":"ContainerStarted","Data":"edffb6bfcbf03f3581e86a18a899cdc6d6d7d15badd4328f7f38ccc66c797ae3"}
Sep 16 19:01:38.116309 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:38.116143    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5" event={"ID":"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d","Type":"ContainerStarted","Data":"bd010e4b8e0d8490c4d518eb634c48f42ad4673ea62768995f535c94c86d291b"}
Sep 16 19:01:38.118341 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:38.118312    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z" event={"ID":"eead0240-eac0-44df-b985-821bce775fbb","Type":"ContainerStarted","Data":"32c8d72af66541c8a8dda78ed752eb8806b2e452210e8984cbf9f25f91fd00d4"}
Sep 16 19:01:38.118483 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:38.118362    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z" event={"ID":"eead0240-eac0-44df-b985-821bce775fbb","Type":"ContainerStarted","Data":"5d4324143812d53ad450aadba0f502d4139b8834854c7559bf6828b7095f755d"}
Sep 16 19:01:38.118624 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:38.118606    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"
Sep 16 19:01:38.118708 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:38.118665    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"
Sep 16 19:01:38.160635 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:38.160582    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z" podStartSLOduration=2.247085273 podStartE2EDuration="6.160564941s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.913479668s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:01:38.159114751 +0000 UTC m=+1008.234884384" watchObservedRunningTime="2026-09-16 19:01:38.160564941 +0000 UTC m=+1008.236334572"
Sep 16 19:01:38.655053 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:38.655013    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 16 19:01:42.141365 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:42.141330    2554 generic.go:356] "Generic (PLEG): container finished" podID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerID="1b4fabf3e8a4ee7e6bd8b5d2ab072c9390e3abb9402108d30e3d077c9267f907" exitCode=0
Sep 16 19:01:42.141763 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:42.141386    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" event={"ID":"099632f8-19f3-4373-b42c-c5cd508b58fc","Type":"ContainerDied","Data":"1b4fabf3e8a4ee7e6bd8b5d2ab072c9390e3abb9402108d30e3d077c9267f907"}
Sep 16 19:01:43.655044 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:43.655000    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 16 19:01:43.660413 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:43.660382    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:01:43.661872 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:43.661843    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:01:44.129173 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:44.129107    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"
Sep 16 19:01:45.158816 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:45.158781    2554 generic.go:356] "Generic (PLEG): container finished" podID="e4b77abe-3eb5-41dc-9e3e-f08f8a98537d" containerID="edffb6bfcbf03f3581e86a18a899cdc6d6d7d15badd4328f7f38ccc66c797ae3" exitCode=0
Sep 16 19:01:45.159324 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:45.158860    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5" event={"ID":"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d","Type":"ContainerDied","Data":"edffb6bfcbf03f3581e86a18a899cdc6d6d7d15badd4328f7f38ccc66c797ae3"}
Sep 16 19:01:45.160291 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:45.160274    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:01:48.655123 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:48.655032    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 16 19:01:48.655697 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:48.655269    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"
Sep 16 19:01:53.654629 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:53.654462    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 16 19:01:53.660416 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:53.660375    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:01:53.662153 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:53.662114    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:01:58.654368 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:01:58.654317    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 16 19:02:03.655305 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:03.655260    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Sep 16 19:02:03.660833 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:03.660789    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Sep 16 19:02:03.661011 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:03.660965    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"
Sep 16 19:02:03.661887 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:03.661857    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 16 19:02:03.662018 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:03.661968    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"
Sep 16 19:02:07.305064 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:07.305017    2554 generic.go:356] "Generic (PLEG): container finished" podID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerID="53e1e563bfddd449ae6107f4a58fb5f581112c65255e4db23e330add1777cefb" exitCode=0
Sep 16 19:02:07.305485 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:07.305086    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" event={"ID":"099632f8-19f3-4373-b42c-c5cd508b58fc","Type":"ContainerDied","Data":"53e1e563bfddd449ae6107f4a58fb5f581112c65255e4db23e330add1777cefb"}
Sep 16 19:02:07.305485 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:07.305173    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7" event={"ID":"099632f8-19f3-4373-b42c-c5cd508b58fc","Type":"ContainerDied","Data":"e90c069c7c96010973981921de491056f5ab4c1c4d5f0532bcc552b489b8dba5"}
Sep 16 19:02:07.305485 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:07.305218    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e90c069c7c96010973981921de491056f5ab4c1c4d5f0532bcc552b489b8dba5"
Sep 16 19:02:07.308379 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:07.308348    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"
Sep 16 19:02:07.445617 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:07.445523    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/099632f8-19f3-4373-b42c-c5cd508b58fc-kube-rbac-proxy-sar-config\") pod \"099632f8-19f3-4373-b42c-c5cd508b58fc\" (UID: \"099632f8-19f3-4373-b42c-c5cd508b58fc\") "
Sep 16 19:02:07.445617 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:07.445568    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/099632f8-19f3-4373-b42c-c5cd508b58fc-kserve-provision-location\") pod \"099632f8-19f3-4373-b42c-c5cd508b58fc\" (UID: \"099632f8-19f3-4373-b42c-c5cd508b58fc\") "
Sep 16 19:02:07.445617 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:07.445595    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/099632f8-19f3-4373-b42c-c5cd508b58fc-cabundle-cert\") pod \"099632f8-19f3-4373-b42c-c5cd508b58fc\" (UID: \"099632f8-19f3-4373-b42c-c5cd508b58fc\") "
Sep 16 19:02:07.445973 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:07.445695    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlsdr\" (UniqueName: \"kubernetes.io/projected/099632f8-19f3-4373-b42c-c5cd508b58fc-kube-api-access-dlsdr\") pod \"099632f8-19f3-4373-b42c-c5cd508b58fc\" (UID: \"099632f8-19f3-4373-b42c-c5cd508b58fc\") "
Sep 16 19:02:07.445973 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:07.445782    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/099632f8-19f3-4373-b42c-c5cd508b58fc-proxy-tls\") pod \"099632f8-19f3-4373-b42c-c5cd508b58fc\" (UID: \"099632f8-19f3-4373-b42c-c5cd508b58fc\") "
Sep 16 19:02:07.445973 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:07.445939    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/099632f8-19f3-4373-b42c-c5cd508b58fc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "099632f8-19f3-4373-b42c-c5cd508b58fc" (UID: "099632f8-19f3-4373-b42c-c5cd508b58fc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:02:07.446130 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:07.445981    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/099632f8-19f3-4373-b42c-c5cd508b58fc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "099632f8-19f3-4373-b42c-c5cd508b58fc" (UID: "099632f8-19f3-4373-b42c-c5cd508b58fc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:07.446130 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:07.446041    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/099632f8-19f3-4373-b42c-c5cd508b58fc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "099632f8-19f3-4373-b42c-c5cd508b58fc" (UID: "099632f8-19f3-4373-b42c-c5cd508b58fc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:02:07.446130 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:07.446102    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/099632f8-19f3-4373-b42c-c5cd508b58fc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:02:07.446130 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:07.446119    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/099632f8-19f3-4373-b42c-c5cd508b58fc-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:02:07.448051 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:07.448026    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/099632f8-19f3-4373-b42c-c5cd508b58fc-kube-api-access-dlsdr" (OuterVolumeSpecName: "kube-api-access-dlsdr") pod "099632f8-19f3-4373-b42c-c5cd508b58fc" (UID: "099632f8-19f3-4373-b42c-c5cd508b58fc"). InnerVolumeSpecName "kube-api-access-dlsdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:02:07.448154 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:07.448107    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/099632f8-19f3-4373-b42c-c5cd508b58fc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "099632f8-19f3-4373-b42c-c5cd508b58fc" (UID: "099632f8-19f3-4373-b42c-c5cd508b58fc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:02:07.547565 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:07.547519    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlsdr\" (UniqueName: \"kubernetes.io/projected/099632f8-19f3-4373-b42c-c5cd508b58fc-kube-api-access-dlsdr\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:02:07.547565 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:07.547554    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/099632f8-19f3-4373-b42c-c5cd508b58fc-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:02:07.547565 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:07.547570    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/099632f8-19f3-4373-b42c-c5cd508b58fc-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:02:08.310223 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:08.310193    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"
Sep 16 19:02:08.336174 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:08.336141    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"]
Sep 16 19:02:08.340314 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:08.340278    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-768b5b646f-tshh7"]
Sep 16 19:02:08.446138 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:08.446103    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" path="/var/lib/kubelet/pods/099632f8-19f3-4373-b42c-c5cd508b58fc/volumes"
Sep 16 19:02:14.130426 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:14.130386    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z" podUID="eead0240-eac0-44df-b985-821bce775fbb" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.36:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:02:24.129818 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:24.129772    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z" podUID="eead0240-eac0-44df-b985-821bce775fbb" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.36:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:02:34.130120 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:34.130082    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z" podUID="eead0240-eac0-44df-b985-821bce775fbb" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.36:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:02:44.129978 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:44.129896    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z" podUID="eead0240-eac0-44df-b985-821bce775fbb" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.36:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:02:54.129797 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:54.129747    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z" podUID="eead0240-eac0-44df-b985-821bce775fbb" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.36:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:02:54.438782 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:02:54.438687    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z" podUID="eead0240-eac0-44df-b985-821bce775fbb" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.36:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:03:04.438759 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:04.438696    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z" podUID="eead0240-eac0-44df-b985-821bce775fbb" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.36:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:03:14.457689 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:14.457653    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"
Sep 16 19:03:22.062876 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:22.062845    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"]
Sep 16 19:03:22.063413 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:22.063082    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z" podUID="eead0240-eac0-44df-b985-821bce775fbb" containerName="kserve-container" containerID="cri-o://5d4324143812d53ad450aadba0f502d4139b8834854c7559bf6828b7095f755d" gracePeriod=30
Sep 16 19:03:22.063413 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:22.063143    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z" podUID="eead0240-eac0-44df-b985-821bce775fbb" containerName="kube-rbac-proxy" containerID="cri-o://32c8d72af66541c8a8dda78ed752eb8806b2e452210e8984cbf9f25f91fd00d4" gracePeriod=30
Sep 16 19:03:24.124161 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:24.124115    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z" podUID="eead0240-eac0-44df-b985-821bce775fbb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 16 19:03:24.439356 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:24.439269    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z" podUID="eead0240-eac0-44df-b985-821bce775fbb" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.36:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:03:29.123895 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:29.123835    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z" podUID="eead0240-eac0-44df-b985-821bce775fbb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 16 19:03:34.124096 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:34.124055    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z" podUID="eead0240-eac0-44df-b985-821bce775fbb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 16 19:03:34.124655 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:34.124180    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"
Sep 16 19:03:34.439092 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:34.439008    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z" podUID="eead0240-eac0-44df-b985-821bce775fbb" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.36:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.134.0.36:8080: connect: connection refused"
Sep 16 19:03:39.124065 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:39.124009    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z" podUID="eead0240-eac0-44df-b985-821bce775fbb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Sep 16 19:03:41.312229 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.312204    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"
Sep 16 19:03:41.406005 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.405938    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eead0240-eac0-44df-b985-821bce775fbb-proxy-tls\") pod \"eead0240-eac0-44df-b985-821bce775fbb\" (UID: \"eead0240-eac0-44df-b985-821bce775fbb\") "
Sep 16 19:03:41.406155 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.406008    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eead0240-eac0-44df-b985-821bce775fbb-kube-rbac-proxy-sar-config\") pod \"eead0240-eac0-44df-b985-821bce775fbb\" (UID: \"eead0240-eac0-44df-b985-821bce775fbb\") "
Sep 16 19:03:41.406155 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.406087    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eead0240-eac0-44df-b985-821bce775fbb-kserve-provision-location\") pod \"eead0240-eac0-44df-b985-821bce775fbb\" (UID: \"eead0240-eac0-44df-b985-821bce775fbb\") "
Sep 16 19:03:41.406155 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.406128    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxscp\" (UniqueName: \"kubernetes.io/projected/eead0240-eac0-44df-b985-821bce775fbb-kube-api-access-bxscp\") pod \"eead0240-eac0-44df-b985-821bce775fbb\" (UID: \"eead0240-eac0-44df-b985-821bce775fbb\") "
Sep 16 19:03:41.406322 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.406209    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eead0240-eac0-44df-b985-821bce775fbb-cabundle-cert\") pod \"eead0240-eac0-44df-b985-821bce775fbb\" (UID: \"eead0240-eac0-44df-b985-821bce775fbb\") "
Sep 16 19:03:41.406383 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.406334    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eead0240-eac0-44df-b985-821bce775fbb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eead0240-eac0-44df-b985-821bce775fbb" (UID: "eead0240-eac0-44df-b985-821bce775fbb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:41.406439 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.406375    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eead0240-eac0-44df-b985-821bce775fbb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eead0240-eac0-44df-b985-821bce775fbb" (UID: "eead0240-eac0-44df-b985-821bce775fbb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:03:41.406596 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.406576    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eead0240-eac0-44df-b985-821bce775fbb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:03:41.406596 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.406597    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eead0240-eac0-44df-b985-821bce775fbb-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:03:41.406736 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.406667    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eead0240-eac0-44df-b985-821bce775fbb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eead0240-eac0-44df-b985-821bce775fbb" (UID: "eead0240-eac0-44df-b985-821bce775fbb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:03:41.408115 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.408090    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eead0240-eac0-44df-b985-821bce775fbb-kube-api-access-bxscp" (OuterVolumeSpecName: "kube-api-access-bxscp") pod "eead0240-eac0-44df-b985-821bce775fbb" (UID: "eead0240-eac0-44df-b985-821bce775fbb"). InnerVolumeSpecName "kube-api-access-bxscp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:03:41.408287 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.408252    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eead0240-eac0-44df-b985-821bce775fbb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eead0240-eac0-44df-b985-821bce775fbb" (UID: "eead0240-eac0-44df-b985-821bce775fbb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:03:41.507895 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.507873    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxscp\" (UniqueName: \"kubernetes.io/projected/eead0240-eac0-44df-b985-821bce775fbb-kube-api-access-bxscp\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:03:41.507895 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.507894    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eead0240-eac0-44df-b985-821bce775fbb-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:03:41.508043 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.507903    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eead0240-eac0-44df-b985-821bce775fbb-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:03:41.829376 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.829347    2554 generic.go:356] "Generic (PLEG): container finished" podID="eead0240-eac0-44df-b985-821bce775fbb" containerID="32c8d72af66541c8a8dda78ed752eb8806b2e452210e8984cbf9f25f91fd00d4" exitCode=2
Sep 16 19:03:41.829376 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.829372    2554 generic.go:356] "Generic (PLEG): container finished" podID="eead0240-eac0-44df-b985-821bce775fbb" containerID="5d4324143812d53ad450aadba0f502d4139b8834854c7559bf6828b7095f755d" exitCode=0
Sep 16 19:03:41.829553 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.829409    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"
Sep 16 19:03:41.829553 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.829429    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z" event={"ID":"eead0240-eac0-44df-b985-821bce775fbb","Type":"ContainerDied","Data":"32c8d72af66541c8a8dda78ed752eb8806b2e452210e8984cbf9f25f91fd00d4"}
Sep 16 19:03:41.829553 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.829482    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z" event={"ID":"eead0240-eac0-44df-b985-821bce775fbb","Type":"ContainerDied","Data":"5d4324143812d53ad450aadba0f502d4139b8834854c7559bf6828b7095f755d"}
Sep 16 19:03:41.829553 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.829505    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z" event={"ID":"eead0240-eac0-44df-b985-821bce775fbb","Type":"ContainerDied","Data":"f620807c713e4d8b5de4b5486a331d87944941c803b2f06f16d307117aca5ffd"}
Sep 16 19:03:41.829553 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.829531    2554 scope.go:117] "RemoveContainer" containerID="32c8d72af66541c8a8dda78ed752eb8806b2e452210e8984cbf9f25f91fd00d4"
Sep 16 19:03:41.851367 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.851347    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"]
Sep 16 19:03:41.855077 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.855056    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-gfl4z"]
Sep 16 19:03:41.904357 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.904344    2554 scope.go:117] "RemoveContainer" containerID="5d4324143812d53ad450aadba0f502d4139b8834854c7559bf6828b7095f755d"
Sep 16 19:03:41.912309 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.912284    2554 scope.go:117] "RemoveContainer" containerID="9eb7f5d49715911c3369c856b4f62fd0b6ecc40086a5f246eb3af8c661365caf"
Sep 16 19:03:41.919503 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.919489    2554 scope.go:117] "RemoveContainer" containerID="32c8d72af66541c8a8dda78ed752eb8806b2e452210e8984cbf9f25f91fd00d4"
Sep 16 19:03:41.919803 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:03:41.919786    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32c8d72af66541c8a8dda78ed752eb8806b2e452210e8984cbf9f25f91fd00d4\": container with ID starting with 32c8d72af66541c8a8dda78ed752eb8806b2e452210e8984cbf9f25f91fd00d4 not found: ID does not exist" containerID="32c8d72af66541c8a8dda78ed752eb8806b2e452210e8984cbf9f25f91fd00d4"
Sep 16 19:03:41.919858 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.919810    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32c8d72af66541c8a8dda78ed752eb8806b2e452210e8984cbf9f25f91fd00d4"} err="failed to get container status \"32c8d72af66541c8a8dda78ed752eb8806b2e452210e8984cbf9f25f91fd00d4\": rpc error: code = NotFound desc = could not find container \"32c8d72af66541c8a8dda78ed752eb8806b2e452210e8984cbf9f25f91fd00d4\": container with ID starting with 32c8d72af66541c8a8dda78ed752eb8806b2e452210e8984cbf9f25f91fd00d4 not found: ID does not exist"
Sep 16 19:03:41.919858 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.919824    2554 scope.go:117] "RemoveContainer" containerID="5d4324143812d53ad450aadba0f502d4139b8834854c7559bf6828b7095f755d"
Sep 16 19:03:41.920098 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:03:41.920082    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d4324143812d53ad450aadba0f502d4139b8834854c7559bf6828b7095f755d\": container with ID starting with 5d4324143812d53ad450aadba0f502d4139b8834854c7559bf6828b7095f755d not found: ID does not exist" containerID="5d4324143812d53ad450aadba0f502d4139b8834854c7559bf6828b7095f755d"
Sep 16 19:03:41.920145 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.920103    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d4324143812d53ad450aadba0f502d4139b8834854c7559bf6828b7095f755d"} err="failed to get container status \"5d4324143812d53ad450aadba0f502d4139b8834854c7559bf6828b7095f755d\": rpc error: code = NotFound desc = could not find container \"5d4324143812d53ad450aadba0f502d4139b8834854c7559bf6828b7095f755d\": container with ID starting with 5d4324143812d53ad450aadba0f502d4139b8834854c7559bf6828b7095f755d not found: ID does not exist"
Sep 16 19:03:41.920145 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.920124    2554 scope.go:117] "RemoveContainer" containerID="9eb7f5d49715911c3369c856b4f62fd0b6ecc40086a5f246eb3af8c661365caf"
Sep 16 19:03:41.920371 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:03:41.920353    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eb7f5d49715911c3369c856b4f62fd0b6ecc40086a5f246eb3af8c661365caf\": container with ID starting with 9eb7f5d49715911c3369c856b4f62fd0b6ecc40086a5f246eb3af8c661365caf not found: ID does not exist" containerID="9eb7f5d49715911c3369c856b4f62fd0b6ecc40086a5f246eb3af8c661365caf"
Sep 16 19:03:41.920437 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.920377    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eb7f5d49715911c3369c856b4f62fd0b6ecc40086a5f246eb3af8c661365caf"} err="failed to get container status \"9eb7f5d49715911c3369c856b4f62fd0b6ecc40086a5f246eb3af8c661365caf\": rpc error: code = NotFound desc = could not find container \"9eb7f5d49715911c3369c856b4f62fd0b6ecc40086a5f246eb3af8c661365caf\": container with ID starting with 9eb7f5d49715911c3369c856b4f62fd0b6ecc40086a5f246eb3af8c661365caf not found: ID does not exist"
Sep 16 19:03:41.920437 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.920391    2554 scope.go:117] "RemoveContainer" containerID="32c8d72af66541c8a8dda78ed752eb8806b2e452210e8984cbf9f25f91fd00d4"
Sep 16 19:03:41.920641 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.920626    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32c8d72af66541c8a8dda78ed752eb8806b2e452210e8984cbf9f25f91fd00d4"} err="failed to get container status \"32c8d72af66541c8a8dda78ed752eb8806b2e452210e8984cbf9f25f91fd00d4\": rpc error: code = NotFound desc = could not find container \"32c8d72af66541c8a8dda78ed752eb8806b2e452210e8984cbf9f25f91fd00d4\": container with ID starting with 32c8d72af66541c8a8dda78ed752eb8806b2e452210e8984cbf9f25f91fd00d4 not found: ID does not exist"
Sep 16 19:03:41.920687 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.920642    2554 scope.go:117] "RemoveContainer" containerID="5d4324143812d53ad450aadba0f502d4139b8834854c7559bf6828b7095f755d"
Sep 16 19:03:41.920965 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.920948    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d4324143812d53ad450aadba0f502d4139b8834854c7559bf6828b7095f755d"} err="failed to get container status \"5d4324143812d53ad450aadba0f502d4139b8834854c7559bf6828b7095f755d\": rpc error: code = NotFound desc = could not find container \"5d4324143812d53ad450aadba0f502d4139b8834854c7559bf6828b7095f755d\": container with ID starting with 5d4324143812d53ad450aadba0f502d4139b8834854c7559bf6828b7095f755d not found: ID does not exist"
Sep 16 19:03:41.921051 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.920965    2554 scope.go:117] "RemoveContainer" containerID="9eb7f5d49715911c3369c856b4f62fd0b6ecc40086a5f246eb3af8c661365caf"
Sep 16 19:03:41.921217 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:41.921198    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eb7f5d49715911c3369c856b4f62fd0b6ecc40086a5f246eb3af8c661365caf"} err="failed to get container status \"9eb7f5d49715911c3369c856b4f62fd0b6ecc40086a5f246eb3af8c661365caf\": rpc error: code = NotFound desc = could not find container \"9eb7f5d49715911c3369c856b4f62fd0b6ecc40086a5f246eb3af8c661365caf\": container with ID starting with 9eb7f5d49715911c3369c856b4f62fd0b6ecc40086a5f246eb3af8c661365caf not found: ID does not exist"
Sep 16 19:03:42.443439 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:42.443414    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eead0240-eac0-44df-b985-821bce775fbb" path="/var/lib/kubelet/pods/eead0240-eac0-44df-b985-821bce775fbb/volumes"
Sep 16 19:03:42.836167 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:42.836132    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5" event={"ID":"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d","Type":"ContainerStarted","Data":"7d27630942694dc096ee88114bcf6458a72701ac8101dfacbaf002c5d8f1106c"}
Sep 16 19:03:42.836349 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:42.836201    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5" event={"ID":"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d","Type":"ContainerStarted","Data":"16e7df991cf884b87fe50f5c4670a6cbfb3477e785670c35ae04e927cbf538cb"}
Sep 16 19:03:42.836349 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:42.836232    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"
Sep 16 19:03:42.836349 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:42.836257    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"
Sep 16 19:03:42.837848 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:42.837819    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5" podUID="e4b77abe-3eb5-41dc-9e3e-f08f8a98537d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:03:42.854340 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:42.854283    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5" podStartSLOduration=3.446105994 podStartE2EDuration="2m6.854264823s" totalImagesPullingTime="1m57.088904305s" totalInitContainerRuntime="6.319254524s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:01:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:03:42.854217748 +0000 UTC m=+1132.929987417" watchObservedRunningTime="2026-09-16 19:03:42.854264823 +0000 UTC m=+1132.930034457"
Sep 16 19:03:43.842235 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:43.842196    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5" podUID="e4b77abe-3eb5-41dc-9e3e-f08f8a98537d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:03:48.846708 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:48.846682    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"
Sep 16 19:03:48.847293 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:48.847267    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5" podUID="e4b77abe-3eb5-41dc-9e3e-f08f8a98537d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Sep 16 19:03:58.848767 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:03:58.848739    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"
Sep 16 19:04:12.546652 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:12.546622    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"]
Sep 16 19:04:12.547100 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:12.546859    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5" podUID="e4b77abe-3eb5-41dc-9e3e-f08f8a98537d" containerName="kserve-container" containerID="cri-o://16e7df991cf884b87fe50f5c4670a6cbfb3477e785670c35ae04e927cbf538cb" gracePeriod=30
Sep 16 19:04:12.547100 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:12.546904    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5" podUID="e4b77abe-3eb5-41dc-9e3e-f08f8a98537d" containerName="kube-rbac-proxy" containerID="cri-o://7d27630942694dc096ee88114bcf6458a72701ac8101dfacbaf002c5d8f1106c" gracePeriod=30
Sep 16 19:04:12.974696 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:12.974583    2554 generic.go:356] "Generic (PLEG): container finished" podID="e4b77abe-3eb5-41dc-9e3e-f08f8a98537d" containerID="7d27630942694dc096ee88114bcf6458a72701ac8101dfacbaf002c5d8f1106c" exitCode=2
Sep 16 19:04:12.974696 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:12.974651    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5" event={"ID":"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d","Type":"ContainerDied","Data":"7d27630942694dc096ee88114bcf6458a72701ac8101dfacbaf002c5d8f1106c"}
Sep 16 19:04:13.822763 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:13.822736    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"
Sep 16 19:04:13.945695 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:13.945640    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-kserve-provision-location\") pod \"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d\" (UID: \"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d\") "
Sep 16 19:04:13.945695 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:13.945684    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbp65\" (UniqueName: \"kubernetes.io/projected/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-kube-api-access-zbp65\") pod \"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d\" (UID: \"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d\") "
Sep 16 19:04:13.945902 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:13.945709    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-kube-rbac-proxy-sar-config\") pod \"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d\" (UID: \"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d\") "
Sep 16 19:04:13.945902 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:13.945744    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-proxy-tls\") pod \"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d\" (UID: \"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d\") "
Sep 16 19:04:13.945902 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:13.945810    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-cabundle-cert\") pod \"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d\" (UID: \"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d\") "
Sep 16 19:04:13.946159 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:13.946128    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e4b77abe-3eb5-41dc-9e3e-f08f8a98537d" (UID: "e4b77abe-3eb5-41dc-9e3e-f08f8a98537d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:04:13.946321 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:13.946187    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e4b77abe-3eb5-41dc-9e3e-f08f8a98537d" (UID: "e4b77abe-3eb5-41dc-9e3e-f08f8a98537d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:04:13.947799 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:13.947773    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-kube-api-access-zbp65" (OuterVolumeSpecName: "kube-api-access-zbp65") pod "e4b77abe-3eb5-41dc-9e3e-f08f8a98537d" (UID: "e4b77abe-3eb5-41dc-9e3e-f08f8a98537d"). InnerVolumeSpecName "kube-api-access-zbp65". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:04:13.948193 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:13.948169    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e4b77abe-3eb5-41dc-9e3e-f08f8a98537d" (UID: "e4b77abe-3eb5-41dc-9e3e-f08f8a98537d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:04:13.950756 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:13.950732    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e4b77abe-3eb5-41dc-9e3e-f08f8a98537d" (UID: "e4b77abe-3eb5-41dc-9e3e-f08f8a98537d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:04:13.981181 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:13.981162    2554 generic.go:356] "Generic (PLEG): container finished" podID="e4b77abe-3eb5-41dc-9e3e-f08f8a98537d" containerID="16e7df991cf884b87fe50f5c4670a6cbfb3477e785670c35ae04e927cbf538cb" exitCode=0
Sep 16 19:04:13.981258 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:13.981206    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5" event={"ID":"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d","Type":"ContainerDied","Data":"16e7df991cf884b87fe50f5c4670a6cbfb3477e785670c35ae04e927cbf538cb"}
Sep 16 19:04:13.981258 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:13.981213    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"
Sep 16 19:04:13.981258 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:13.981248    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5" event={"ID":"e4b77abe-3eb5-41dc-9e3e-f08f8a98537d","Type":"ContainerDied","Data":"bd010e4b8e0d8490c4d518eb634c48f42ad4673ea62768995f535c94c86d291b"}
Sep 16 19:04:13.981359 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:13.981273    2554 scope.go:117] "RemoveContainer" containerID="7d27630942694dc096ee88114bcf6458a72701ac8101dfacbaf002c5d8f1106c"
Sep 16 19:04:13.989272 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:13.989250    2554 scope.go:117] "RemoveContainer" containerID="16e7df991cf884b87fe50f5c4670a6cbfb3477e785670c35ae04e927cbf538cb"
Sep 16 19:04:13.996471 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:13.996454    2554 scope.go:117] "RemoveContainer" containerID="edffb6bfcbf03f3581e86a18a899cdc6d6d7d15badd4328f7f38ccc66c797ae3"
Sep 16 19:04:14.001951 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:14.001929    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"]
Sep 16 19:04:14.003992 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:14.003966    2554 scope.go:117] "RemoveContainer" containerID="7d27630942694dc096ee88114bcf6458a72701ac8101dfacbaf002c5d8f1106c"
Sep 16 19:04:14.004327 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:04:14.004298    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d27630942694dc096ee88114bcf6458a72701ac8101dfacbaf002c5d8f1106c\": container with ID starting with 7d27630942694dc096ee88114bcf6458a72701ac8101dfacbaf002c5d8f1106c not found: ID does not exist" containerID="7d27630942694dc096ee88114bcf6458a72701ac8101dfacbaf002c5d8f1106c"
Sep 16 19:04:14.004491 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:14.004333    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d27630942694dc096ee88114bcf6458a72701ac8101dfacbaf002c5d8f1106c"} err="failed to get container status \"7d27630942694dc096ee88114bcf6458a72701ac8101dfacbaf002c5d8f1106c\": rpc error: code = NotFound desc = could not find container \"7d27630942694dc096ee88114bcf6458a72701ac8101dfacbaf002c5d8f1106c\": container with ID starting with 7d27630942694dc096ee88114bcf6458a72701ac8101dfacbaf002c5d8f1106c not found: ID does not exist"
Sep 16 19:04:14.004491 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:14.004348    2554 scope.go:117] "RemoveContainer" containerID="16e7df991cf884b87fe50f5c4670a6cbfb3477e785670c35ae04e927cbf538cb"
Sep 16 19:04:14.004841 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:04:14.004815    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16e7df991cf884b87fe50f5c4670a6cbfb3477e785670c35ae04e927cbf538cb\": container with ID starting with 16e7df991cf884b87fe50f5c4670a6cbfb3477e785670c35ae04e927cbf538cb not found: ID does not exist" containerID="16e7df991cf884b87fe50f5c4670a6cbfb3477e785670c35ae04e927cbf538cb"
Sep 16 19:04:14.004923 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:14.004840    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16e7df991cf884b87fe50f5c4670a6cbfb3477e785670c35ae04e927cbf538cb"} err="failed to get container status \"16e7df991cf884b87fe50f5c4670a6cbfb3477e785670c35ae04e927cbf538cb\": rpc error: code = NotFound desc = could not find container \"16e7df991cf884b87fe50f5c4670a6cbfb3477e785670c35ae04e927cbf538cb\": container with ID starting with 16e7df991cf884b87fe50f5c4670a6cbfb3477e785670c35ae04e927cbf538cb not found: ID does not exist"
Sep 16 19:04:14.004923 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:14.004856    2554 scope.go:117] "RemoveContainer" containerID="edffb6bfcbf03f3581e86a18a899cdc6d6d7d15badd4328f7f38ccc66c797ae3"
Sep 16 19:04:14.005151 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:04:14.005136    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edffb6bfcbf03f3581e86a18a899cdc6d6d7d15badd4328f7f38ccc66c797ae3\": container with ID starting with edffb6bfcbf03f3581e86a18a899cdc6d6d7d15badd4328f7f38ccc66c797ae3 not found: ID does not exist" containerID="edffb6bfcbf03f3581e86a18a899cdc6d6d7d15badd4328f7f38ccc66c797ae3"
Sep 16 19:04:14.005203 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:14.005154    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edffb6bfcbf03f3581e86a18a899cdc6d6d7d15badd4328f7f38ccc66c797ae3"} err="failed to get container status \"edffb6bfcbf03f3581e86a18a899cdc6d6d7d15badd4328f7f38ccc66c797ae3\": rpc error: code = NotFound desc = could not find container \"edffb6bfcbf03f3581e86a18a899cdc6d6d7d15badd4328f7f38ccc66c797ae3\": container with ID starting with edffb6bfcbf03f3581e86a18a899cdc6d6d7d15badd4328f7f38ccc66c797ae3 not found: ID does not exist"
Sep 16 19:04:14.006239 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:14.006221    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-574b5b7495-2b5c5"]
Sep 16 19:04:14.046769 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:14.046748    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:04:14.046769 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:14.046768    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbp65\" (UniqueName: \"kubernetes.io/projected/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-kube-api-access-zbp65\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:04:14.046878 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:14.046778    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:04:14.046878 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:14.046787    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:04:14.046878 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:14.046798    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:04:14.447679 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:14.447654    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4b77abe-3eb5-41dc-9e3e-f08f8a98537d" path="/var/lib/kubelet/pods/e4b77abe-3eb5-41dc-9e3e-f08f8a98537d/volumes"
Sep 16 19:04:17.756004 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.755975    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7"]
Sep 16 19:04:17.756437 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.756308    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="storage-initializer"
Sep 16 19:04:17.756437 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.756320    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="storage-initializer"
Sep 16 19:04:17.756437 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.756329    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="kserve-container"
Sep 16 19:04:17.756437 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.756335    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="kserve-container"
Sep 16 19:04:17.756437 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.756342    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="agent"
Sep 16 19:04:17.756437 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.756348    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="agent"
Sep 16 19:04:17.756437 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.756355    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4b77abe-3eb5-41dc-9e3e-f08f8a98537d" containerName="storage-initializer"
Sep 16 19:04:17.756437 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.756360    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4b77abe-3eb5-41dc-9e3e-f08f8a98537d" containerName="storage-initializer"
Sep 16 19:04:17.756437 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.756366    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="kube-rbac-proxy"
Sep 16 19:04:17.756437 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.756371    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="kube-rbac-proxy"
Sep 16 19:04:17.756437 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.756380    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eead0240-eac0-44df-b985-821bce775fbb" containerName="kube-rbac-proxy"
Sep 16 19:04:17.756437 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.756385    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="eead0240-eac0-44df-b985-821bce775fbb" containerName="kube-rbac-proxy"
Sep 16 19:04:17.756437 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.756393    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4b77abe-3eb5-41dc-9e3e-f08f8a98537d" containerName="kube-rbac-proxy"
Sep 16 19:04:17.756437 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.756398    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4b77abe-3eb5-41dc-9e3e-f08f8a98537d" containerName="kube-rbac-proxy"
Sep 16 19:04:17.756437 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.756411    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eead0240-eac0-44df-b985-821bce775fbb" containerName="storage-initializer"
Sep 16 19:04:17.756437 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.756416    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="eead0240-eac0-44df-b985-821bce775fbb" containerName="storage-initializer"
Sep 16 19:04:17.756437 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.756423    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eead0240-eac0-44df-b985-821bce775fbb" containerName="kserve-container"
Sep 16 19:04:17.756437 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.756428    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="eead0240-eac0-44df-b985-821bce775fbb" containerName="kserve-container"
Sep 16 19:04:17.756437 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.756439    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4b77abe-3eb5-41dc-9e3e-f08f8a98537d" containerName="kserve-container"
Sep 16 19:04:17.756437 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.756445    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4b77abe-3eb5-41dc-9e3e-f08f8a98537d" containerName="kserve-container"
Sep 16 19:04:17.757155 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.756513    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="eead0240-eac0-44df-b985-821bce775fbb" containerName="kube-rbac-proxy"
Sep 16 19:04:17.757155 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.756521    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="agent"
Sep 16 19:04:17.757155 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.756527    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4b77abe-3eb5-41dc-9e3e-f08f8a98537d" containerName="kserve-container"
Sep 16 19:04:17.757155 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.756535    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4b77abe-3eb5-41dc-9e3e-f08f8a98537d" containerName="kube-rbac-proxy"
Sep 16 19:04:17.757155 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.756541    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="eead0240-eac0-44df-b985-821bce775fbb" containerName="kserve-container"
Sep 16 19:04:17.757155 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.756547    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="kube-rbac-proxy"
Sep 16 19:04:17.757155 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.756553    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="099632f8-19f3-4373-b42c-c5cd508b58fc" containerName="kserve-container"
Sep 16 19:04:17.759642 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.759626    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7"
Sep 16 19:04:17.762067 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.762044    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:04:17.762694 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.762675    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Sep 16 19:04:17.762793 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.762686    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8gklb\""
Sep 16 19:04:17.762859 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.762732    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Sep 16 19:04:17.762918 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.762882    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:04:17.762974 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.762915    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:04:17.763029 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.762984    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:04:17.772099 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.772078    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7"]
Sep 16 19:04:17.867843 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.867818    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db9efbe7-ac5c-451d-a93a-fba78bb09021-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-prbn7\" (UID: \"db9efbe7-ac5c-451d-a93a-fba78bb09021\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7"
Sep 16 19:04:17.867933 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.867854    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db9efbe7-ac5c-451d-a93a-fba78bb09021-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-prbn7\" (UID: \"db9efbe7-ac5c-451d-a93a-fba78bb09021\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7"
Sep 16 19:04:17.867933 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.867874    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd7kn\" (UniqueName: \"kubernetes.io/projected/db9efbe7-ac5c-451d-a93a-fba78bb09021-kube-api-access-fd7kn\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-prbn7\" (UID: \"db9efbe7-ac5c-451d-a93a-fba78bb09021\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7"
Sep 16 19:04:17.868018 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.867954    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db9efbe7-ac5c-451d-a93a-fba78bb09021-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-prbn7\" (UID: \"db9efbe7-ac5c-451d-a93a-fba78bb09021\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7"
Sep 16 19:04:17.868018 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.867981    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db9efbe7-ac5c-451d-a93a-fba78bb09021-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-prbn7\" (UID: \"db9efbe7-ac5c-451d-a93a-fba78bb09021\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7"
Sep 16 19:04:17.969064 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.969041    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db9efbe7-ac5c-451d-a93a-fba78bb09021-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-prbn7\" (UID: \"db9efbe7-ac5c-451d-a93a-fba78bb09021\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7"
Sep 16 19:04:17.969157 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.969067    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd7kn\" (UniqueName: \"kubernetes.io/projected/db9efbe7-ac5c-451d-a93a-fba78bb09021-kube-api-access-fd7kn\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-prbn7\" (UID: \"db9efbe7-ac5c-451d-a93a-fba78bb09021\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7"
Sep 16 19:04:17.969157 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.969101    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db9efbe7-ac5c-451d-a93a-fba78bb09021-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-prbn7\" (UID: \"db9efbe7-ac5c-451d-a93a-fba78bb09021\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7"
Sep 16 19:04:17.969157 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.969119    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db9efbe7-ac5c-451d-a93a-fba78bb09021-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-prbn7\" (UID: \"db9efbe7-ac5c-451d-a93a-fba78bb09021\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7"
Sep 16 19:04:17.969157 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.969155    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db9efbe7-ac5c-451d-a93a-fba78bb09021-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-prbn7\" (UID: \"db9efbe7-ac5c-451d-a93a-fba78bb09021\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7"
Sep 16 19:04:17.969515 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.969496    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db9efbe7-ac5c-451d-a93a-fba78bb09021-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-prbn7\" (UID: \"db9efbe7-ac5c-451d-a93a-fba78bb09021\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7"
Sep 16 19:04:17.969647 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.969636    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db9efbe7-ac5c-451d-a93a-fba78bb09021-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-prbn7\" (UID: \"db9efbe7-ac5c-451d-a93a-fba78bb09021\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7"
Sep 16 19:04:17.969846 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.969826    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db9efbe7-ac5c-451d-a93a-fba78bb09021-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-prbn7\" (UID: \"db9efbe7-ac5c-451d-a93a-fba78bb09021\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7"
Sep 16 19:04:17.971292 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.971267    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db9efbe7-ac5c-451d-a93a-fba78bb09021-proxy-tls\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-prbn7\" (UID: \"db9efbe7-ac5c-451d-a93a-fba78bb09021\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7"
Sep 16 19:04:17.977690 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:17.977672    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd7kn\" (UniqueName: \"kubernetes.io/projected/db9efbe7-ac5c-451d-a93a-fba78bb09021-kube-api-access-fd7kn\") pod \"isvc-autogluon-v2-predictor-9684b4cd9-prbn7\" (UID: \"db9efbe7-ac5c-451d-a93a-fba78bb09021\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7"
Sep 16 19:04:18.069636 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:18.069587    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7"
Sep 16 19:04:18.340710 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:18.340690    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7"]
Sep 16 19:04:18.342882 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:04:18.342860    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb9efbe7_ac5c_451d_a93a_fba78bb09021.slice/crio-7816354d768b2e6c89510370db774e79687b6cae459cec0422d9e6f9e94fe151 WatchSource:0}: Error finding container 7816354d768b2e6c89510370db774e79687b6cae459cec0422d9e6f9e94fe151: Status 404 returned error can't find the container with id 7816354d768b2e6c89510370db774e79687b6cae459cec0422d9e6f9e94fe151
Sep 16 19:04:19.009027 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:19.008993    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7" event={"ID":"db9efbe7-ac5c-451d-a93a-fba78bb09021","Type":"ContainerStarted","Data":"bdd4de2940f6756c9d66371a4c37377d6779aee62d85b75050b70fd968fb92d4"}
Sep 16 19:04:19.009397 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:19.009048    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7" event={"ID":"db9efbe7-ac5c-451d-a93a-fba78bb09021","Type":"ContainerStarted","Data":"7816354d768b2e6c89510370db774e79687b6cae459cec0422d9e6f9e94fe151"}
Sep 16 19:04:24.033268 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:24.033235    2554 generic.go:356] "Generic (PLEG): container finished" podID="db9efbe7-ac5c-451d-a93a-fba78bb09021" containerID="bdd4de2940f6756c9d66371a4c37377d6779aee62d85b75050b70fd968fb92d4" exitCode=0
Sep 16 19:04:24.033268 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:24.033270    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7" event={"ID":"db9efbe7-ac5c-451d-a93a-fba78bb09021","Type":"ContainerDied","Data":"bdd4de2940f6756c9d66371a4c37377d6779aee62d85b75050b70fd968fb92d4"}
Sep 16 19:04:25.040044 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:25.040014    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7" event={"ID":"db9efbe7-ac5c-451d-a93a-fba78bb09021","Type":"ContainerStarted","Data":"e32320d49bfb54c0234365c19ac0f70bf5c8206b9d08b9c8d386334d6e7cbeb4"}
Sep 16 19:04:25.040378 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:25.040070    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7" event={"ID":"db9efbe7-ac5c-451d-a93a-fba78bb09021","Type":"ContainerStarted","Data":"03c17ec05f5b6f5d9c5553eb5a41e7723df0fb190a596b8a84fb2ae9e87fa693"}
Sep 16 19:04:25.040378 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:25.040268    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7"
Sep 16 19:04:25.060385 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:25.060339    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7" podStartSLOduration=3.070059513 podStartE2EDuration="8.060325857s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.990266344s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:04:25.057200921 +0000 UTC m=+1175.132970574" watchObservedRunningTime="2026-09-16 19:04:25.060325857 +0000 UTC m=+1175.136095487"
Sep 16 19:04:26.045304 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:26.045273    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7"
Sep 16 19:04:32.055038 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:32.055004    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7"
Sep 16 19:04:50.406899 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:50.406862    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/2.log"
Sep 16 19:04:50.408207 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:50.408175    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/2.log"
Sep 16 19:04:50.413906 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:50.413885    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovn-acl-logging/0.log"
Sep 16 19:04:50.414907 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:04:50.414887    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovn-acl-logging/0.log"
Sep 16 19:06:00.444307 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:00.444235    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7"
Sep 16 19:06:11.481921 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:11.481878    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7" podUID="db9efbe7-ac5c-451d-a93a-fba78bb09021" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.38:8080/v2/models/isvc-autogluon-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:06:13.757042 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:13.757009    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7"]
Sep 16 19:06:13.757511 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:13.757333    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7" podUID="db9efbe7-ac5c-451d-a93a-fba78bb09021" containerName="kserve-container" containerID="cri-o://03c17ec05f5b6f5d9c5553eb5a41e7723df0fb190a596b8a84fb2ae9e87fa693" gracePeriod=30
Sep 16 19:06:13.757511 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:13.757385    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7" podUID="db9efbe7-ac5c-451d-a93a-fba78bb09021" containerName="kube-rbac-proxy" containerID="cri-o://e32320d49bfb54c0234365c19ac0f70bf5c8206b9d08b9c8d386334d6e7cbeb4" gracePeriod=30
Sep 16 19:06:14.549604 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:14.549571    2554 generic.go:356] "Generic (PLEG): container finished" podID="db9efbe7-ac5c-451d-a93a-fba78bb09021" containerID="e32320d49bfb54c0234365c19ac0f70bf5c8206b9d08b9c8d386334d6e7cbeb4" exitCode=2
Sep 16 19:06:14.549825 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:14.549622    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7" event={"ID":"db9efbe7-ac5c-451d-a93a-fba78bb09021","Type":"ContainerDied","Data":"e32320d49bfb54c0234365c19ac0f70bf5c8206b9d08b9c8d386334d6e7cbeb4"}
Sep 16 19:06:15.006901 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.006863    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7"
Sep 16 19:06:15.096145 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.096120    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db9efbe7-ac5c-451d-a93a-fba78bb09021-kube-rbac-proxy-sar-config\") pod \"db9efbe7-ac5c-451d-a93a-fba78bb09021\" (UID: \"db9efbe7-ac5c-451d-a93a-fba78bb09021\") "
Sep 16 19:06:15.096268 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.096180    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db9efbe7-ac5c-451d-a93a-fba78bb09021-kserve-provision-location\") pod \"db9efbe7-ac5c-451d-a93a-fba78bb09021\" (UID: \"db9efbe7-ac5c-451d-a93a-fba78bb09021\") "
Sep 16 19:06:15.096268 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.096198    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db9efbe7-ac5c-451d-a93a-fba78bb09021-proxy-tls\") pod \"db9efbe7-ac5c-451d-a93a-fba78bb09021\" (UID: \"db9efbe7-ac5c-451d-a93a-fba78bb09021\") "
Sep 16 19:06:15.096268 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.096260    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db9efbe7-ac5c-451d-a93a-fba78bb09021-cabundle-cert\") pod \"db9efbe7-ac5c-451d-a93a-fba78bb09021\" (UID: \"db9efbe7-ac5c-451d-a93a-fba78bb09021\") "
Sep 16 19:06:15.096413 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.096280    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fd7kn\" (UniqueName: \"kubernetes.io/projected/db9efbe7-ac5c-451d-a93a-fba78bb09021-kube-api-access-fd7kn\") pod \"db9efbe7-ac5c-451d-a93a-fba78bb09021\" (UID: \"db9efbe7-ac5c-451d-a93a-fba78bb09021\") "
Sep 16 19:06:15.096467 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.096438    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db9efbe7-ac5c-451d-a93a-fba78bb09021-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "db9efbe7-ac5c-451d-a93a-fba78bb09021" (UID: "db9efbe7-ac5c-451d-a93a-fba78bb09021"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:06:15.096677 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.096650    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db9efbe7-ac5c-451d-a93a-fba78bb09021-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "db9efbe7-ac5c-451d-a93a-fba78bb09021" (UID: "db9efbe7-ac5c-451d-a93a-fba78bb09021"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:06:15.098418 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.098316    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db9efbe7-ac5c-451d-a93a-fba78bb09021-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "db9efbe7-ac5c-451d-a93a-fba78bb09021" (UID: "db9efbe7-ac5c-451d-a93a-fba78bb09021"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:06:15.098418 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.098393    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db9efbe7-ac5c-451d-a93a-fba78bb09021-kube-api-access-fd7kn" (OuterVolumeSpecName: "kube-api-access-fd7kn") pod "db9efbe7-ac5c-451d-a93a-fba78bb09021" (UID: "db9efbe7-ac5c-451d-a93a-fba78bb09021"). InnerVolumeSpecName "kube-api-access-fd7kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:06:15.100522 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.100504    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db9efbe7-ac5c-451d-a93a-fba78bb09021-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "db9efbe7-ac5c-451d-a93a-fba78bb09021" (UID: "db9efbe7-ac5c-451d-a93a-fba78bb09021"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:06:15.196949 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.196896    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/db9efbe7-ac5c-451d-a93a-fba78bb09021-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:06:15.196949 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.196915    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/db9efbe7-ac5c-451d-a93a-fba78bb09021-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:06:15.196949 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.196925    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/db9efbe7-ac5c-451d-a93a-fba78bb09021-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:06:15.196949 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.196935    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd7kn\" (UniqueName: \"kubernetes.io/projected/db9efbe7-ac5c-451d-a93a-fba78bb09021-kube-api-access-fd7kn\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:06:15.196949 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.196943    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/db9efbe7-ac5c-451d-a93a-fba78bb09021-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:06:15.556506 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.556474    2554 generic.go:356] "Generic (PLEG): container finished" podID="db9efbe7-ac5c-451d-a93a-fba78bb09021" containerID="03c17ec05f5b6f5d9c5553eb5a41e7723df0fb190a596b8a84fb2ae9e87fa693" exitCode=0
Sep 16 19:06:15.556675 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.556539    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7" event={"ID":"db9efbe7-ac5c-451d-a93a-fba78bb09021","Type":"ContainerDied","Data":"03c17ec05f5b6f5d9c5553eb5a41e7723df0fb190a596b8a84fb2ae9e87fa693"}
Sep 16 19:06:15.556675 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.556548    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7"
Sep 16 19:06:15.556675 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.556586    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7" event={"ID":"db9efbe7-ac5c-451d-a93a-fba78bb09021","Type":"ContainerDied","Data":"7816354d768b2e6c89510370db774e79687b6cae459cec0422d9e6f9e94fe151"}
Sep 16 19:06:15.556675 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.556611    2554 scope.go:117] "RemoveContainer" containerID="e32320d49bfb54c0234365c19ac0f70bf5c8206b9d08b9c8d386334d6e7cbeb4"
Sep 16 19:06:15.565536 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.565515    2554 scope.go:117] "RemoveContainer" containerID="03c17ec05f5b6f5d9c5553eb5a41e7723df0fb190a596b8a84fb2ae9e87fa693"
Sep 16 19:06:15.576390 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.576369    2554 scope.go:117] "RemoveContainer" containerID="bdd4de2940f6756c9d66371a4c37377d6779aee62d85b75050b70fd968fb92d4"
Sep 16 19:06:15.580801 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.580777    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7"]
Sep 16 19:06:15.583356 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.583333    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-9684b4cd9-prbn7"]
Sep 16 19:06:15.584523 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.584496    2554 scope.go:117] "RemoveContainer" containerID="e32320d49bfb54c0234365c19ac0f70bf5c8206b9d08b9c8d386334d6e7cbeb4"
Sep 16 19:06:15.584801 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:06:15.584781    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e32320d49bfb54c0234365c19ac0f70bf5c8206b9d08b9c8d386334d6e7cbeb4\": container with ID starting with e32320d49bfb54c0234365c19ac0f70bf5c8206b9d08b9c8d386334d6e7cbeb4 not found: ID does not exist" containerID="e32320d49bfb54c0234365c19ac0f70bf5c8206b9d08b9c8d386334d6e7cbeb4"
Sep 16 19:06:15.584867 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.584811    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e32320d49bfb54c0234365c19ac0f70bf5c8206b9d08b9c8d386334d6e7cbeb4"} err="failed to get container status \"e32320d49bfb54c0234365c19ac0f70bf5c8206b9d08b9c8d386334d6e7cbeb4\": rpc error: code = NotFound desc = could not find container \"e32320d49bfb54c0234365c19ac0f70bf5c8206b9d08b9c8d386334d6e7cbeb4\": container with ID starting with e32320d49bfb54c0234365c19ac0f70bf5c8206b9d08b9c8d386334d6e7cbeb4 not found: ID does not exist"
Sep 16 19:06:15.584867 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.584829    2554 scope.go:117] "RemoveContainer" containerID="03c17ec05f5b6f5d9c5553eb5a41e7723df0fb190a596b8a84fb2ae9e87fa693"
Sep 16 19:06:15.585069 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:06:15.585051    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03c17ec05f5b6f5d9c5553eb5a41e7723df0fb190a596b8a84fb2ae9e87fa693\": container with ID starting with 03c17ec05f5b6f5d9c5553eb5a41e7723df0fb190a596b8a84fb2ae9e87fa693 not found: ID does not exist" containerID="03c17ec05f5b6f5d9c5553eb5a41e7723df0fb190a596b8a84fb2ae9e87fa693"
Sep 16 19:06:15.585126 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.585073    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03c17ec05f5b6f5d9c5553eb5a41e7723df0fb190a596b8a84fb2ae9e87fa693"} err="failed to get container status \"03c17ec05f5b6f5d9c5553eb5a41e7723df0fb190a596b8a84fb2ae9e87fa693\": rpc error: code = NotFound desc = could not find container \"03c17ec05f5b6f5d9c5553eb5a41e7723df0fb190a596b8a84fb2ae9e87fa693\": container with ID starting with 03c17ec05f5b6f5d9c5553eb5a41e7723df0fb190a596b8a84fb2ae9e87fa693 not found: ID does not exist"
Sep 16 19:06:15.585126 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.585089    2554 scope.go:117] "RemoveContainer" containerID="bdd4de2940f6756c9d66371a4c37377d6779aee62d85b75050b70fd968fb92d4"
Sep 16 19:06:15.585321 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:06:15.585306    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdd4de2940f6756c9d66371a4c37377d6779aee62d85b75050b70fd968fb92d4\": container with ID starting with bdd4de2940f6756c9d66371a4c37377d6779aee62d85b75050b70fd968fb92d4 not found: ID does not exist" containerID="bdd4de2940f6756c9d66371a4c37377d6779aee62d85b75050b70fd968fb92d4"
Sep 16 19:06:15.585378 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:15.585325    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdd4de2940f6756c9d66371a4c37377d6779aee62d85b75050b70fd968fb92d4"} err="failed to get container status \"bdd4de2940f6756c9d66371a4c37377d6779aee62d85b75050b70fd968fb92d4\": rpc error: code = NotFound desc = could not find container \"bdd4de2940f6756c9d66371a4c37377d6779aee62d85b75050b70fd968fb92d4\": container with ID starting with bdd4de2940f6756c9d66371a4c37377d6779aee62d85b75050b70fd968fb92d4 not found: ID does not exist"
Sep 16 19:06:16.443293 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:16.443260    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db9efbe7-ac5c-451d-a93a-fba78bb09021" path="/var/lib/kubelet/pods/db9efbe7-ac5c-451d-a93a-fba78bb09021/volumes"
Sep 16 19:06:18.998136 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:18.998102    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"]
Sep 16 19:06:18.998610 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:18.998576    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db9efbe7-ac5c-451d-a93a-fba78bb09021" containerName="storage-initializer"
Sep 16 19:06:18.998610 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:18.998593    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="db9efbe7-ac5c-451d-a93a-fba78bb09021" containerName="storage-initializer"
Sep 16 19:06:18.998749 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:18.998624    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db9efbe7-ac5c-451d-a93a-fba78bb09021" containerName="kserve-container"
Sep 16 19:06:18.998749 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:18.998632    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="db9efbe7-ac5c-451d-a93a-fba78bb09021" containerName="kserve-container"
Sep 16 19:06:18.998749 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:18.998642    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db9efbe7-ac5c-451d-a93a-fba78bb09021" containerName="kube-rbac-proxy"
Sep 16 19:06:18.998749 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:18.998653    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="db9efbe7-ac5c-451d-a93a-fba78bb09021" containerName="kube-rbac-proxy"
Sep 16 19:06:18.998965 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:18.998756    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="db9efbe7-ac5c-451d-a93a-fba78bb09021" containerName="kube-rbac-proxy"
Sep 16 19:06:18.998965 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:18.998776    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="db9efbe7-ac5c-451d-a93a-fba78bb09021" containerName="kserve-container"
Sep 16 19:06:19.002410 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:19.002390    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"
Sep 16 19:06:19.005021 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:19.004991    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Sep 16 19:06:19.005191 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:19.004991    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:06:19.005191 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:19.005177    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:06:19.005320 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:19.005272    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:06:19.005382 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:19.005317    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8gklb\""
Sep 16 19:06:19.005494 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:19.005451    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Sep 16 19:06:19.006479 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:19.006463    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:06:19.016685 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:19.016663    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"]
Sep 16 19:06:19.028749 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:19.028709    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4803396d-af3a-4d12-8fe2-c37eaf3e6849-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q\" (UID: \"4803396d-af3a-4d12-8fe2-c37eaf3e6849\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"
Sep 16 19:06:19.028837 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:19.028775    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cczz8\" (UniqueName: \"kubernetes.io/projected/4803396d-af3a-4d12-8fe2-c37eaf3e6849-kube-api-access-cczz8\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q\" (UID: \"4803396d-af3a-4d12-8fe2-c37eaf3e6849\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"
Sep 16 19:06:19.028837 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:19.028806    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4803396d-af3a-4d12-8fe2-c37eaf3e6849-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q\" (UID: \"4803396d-af3a-4d12-8fe2-c37eaf3e6849\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"
Sep 16 19:06:19.028837 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:19.028825    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4803396d-af3a-4d12-8fe2-c37eaf3e6849-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q\" (UID: \"4803396d-af3a-4d12-8fe2-c37eaf3e6849\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"
Sep 16 19:06:19.028943 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:19.028861    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4803396d-af3a-4d12-8fe2-c37eaf3e6849-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q\" (UID: \"4803396d-af3a-4d12-8fe2-c37eaf3e6849\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"
Sep 16 19:06:19.129571 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:19.129541    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4803396d-af3a-4d12-8fe2-c37eaf3e6849-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q\" (UID: \"4803396d-af3a-4d12-8fe2-c37eaf3e6849\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"
Sep 16 19:06:19.129711 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:19.129651    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4803396d-af3a-4d12-8fe2-c37eaf3e6849-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q\" (UID: \"4803396d-af3a-4d12-8fe2-c37eaf3e6849\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"
Sep 16 19:06:19.129711 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:19.129680    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cczz8\" (UniqueName: \"kubernetes.io/projected/4803396d-af3a-4d12-8fe2-c37eaf3e6849-kube-api-access-cczz8\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q\" (UID: \"4803396d-af3a-4d12-8fe2-c37eaf3e6849\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"
Sep 16 19:06:19.129711 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:19.129705    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4803396d-af3a-4d12-8fe2-c37eaf3e6849-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q\" (UID: \"4803396d-af3a-4d12-8fe2-c37eaf3e6849\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"
Sep 16 19:06:19.129922 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:19.129753    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4803396d-af3a-4d12-8fe2-c37eaf3e6849-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q\" (UID: \"4803396d-af3a-4d12-8fe2-c37eaf3e6849\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"
Sep 16 19:06:19.129922 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:06:19.129834    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-serving-cert: secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 16 19:06:19.129922 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:06:19.129896    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4803396d-af3a-4d12-8fe2-c37eaf3e6849-proxy-tls podName:4803396d-af3a-4d12-8fe2-c37eaf3e6849 nodeName:}" failed. No retries permitted until 2026-09-16 19:06:19.629881972 +0000 UTC m=+1289.705651582 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4803396d-af3a-4d12-8fe2-c37eaf3e6849-proxy-tls") pod "isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q" (UID: "4803396d-af3a-4d12-8fe2-c37eaf3e6849") : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Sep 16 19:06:19.130098 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:19.130017    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4803396d-af3a-4d12-8fe2-c37eaf3e6849-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q\" (UID: \"4803396d-af3a-4d12-8fe2-c37eaf3e6849\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"
Sep 16 19:06:19.130257 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:19.130237    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4803396d-af3a-4d12-8fe2-c37eaf3e6849-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q\" (UID: \"4803396d-af3a-4d12-8fe2-c37eaf3e6849\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"
Sep 16 19:06:19.130390 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:19.130367    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4803396d-af3a-4d12-8fe2-c37eaf3e6849-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q\" (UID: \"4803396d-af3a-4d12-8fe2-c37eaf3e6849\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"
Sep 16 19:06:19.137983 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:19.137962    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cczz8\" (UniqueName: \"kubernetes.io/projected/4803396d-af3a-4d12-8fe2-c37eaf3e6849-kube-api-access-cczz8\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q\" (UID: \"4803396d-af3a-4d12-8fe2-c37eaf3e6849\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"
Sep 16 19:06:19.634380 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:19.634346    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4803396d-af3a-4d12-8fe2-c37eaf3e6849-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q\" (UID: \"4803396d-af3a-4d12-8fe2-c37eaf3e6849\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"
Sep 16 19:06:19.636759 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:19.636737    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4803396d-af3a-4d12-8fe2-c37eaf3e6849-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q\" (UID: \"4803396d-af3a-4d12-8fe2-c37eaf3e6849\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"
Sep 16 19:06:19.913171 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:19.913077    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"
Sep 16 19:06:20.184464 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:20.184439    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"]
Sep 16 19:06:20.186951 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:06:20.186924    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4803396d_af3a_4d12_8fe2_c37eaf3e6849.slice/crio-d5ffbb9e42a59a93ffb5cc4ef24326cdac3c616ed64176a4260516035aa81e3d WatchSource:0}: Error finding container d5ffbb9e42a59a93ffb5cc4ef24326cdac3c616ed64176a4260516035aa81e3d: Status 404 returned error can't find the container with id d5ffbb9e42a59a93ffb5cc4ef24326cdac3c616ed64176a4260516035aa81e3d
Sep 16 19:06:20.587245 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:20.587206    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q" event={"ID":"4803396d-af3a-4d12-8fe2-c37eaf3e6849","Type":"ContainerStarted","Data":"22e8c3662bcf88f1d632ea88c087a2374f73b054a4a4068f75d3b00fe579aad8"}
Sep 16 19:06:20.587475 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:20.587281    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q" event={"ID":"4803396d-af3a-4d12-8fe2-c37eaf3e6849","Type":"ContainerStarted","Data":"d5ffbb9e42a59a93ffb5cc4ef24326cdac3c616ed64176a4260516035aa81e3d"}
Sep 16 19:06:49.733073 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:49.733037    2554 generic.go:356] "Generic (PLEG): container finished" podID="4803396d-af3a-4d12-8fe2-c37eaf3e6849" containerID="22e8c3662bcf88f1d632ea88c087a2374f73b054a4a4068f75d3b00fe579aad8" exitCode=0
Sep 16 19:06:49.733487 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:49.733106    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q" event={"ID":"4803396d-af3a-4d12-8fe2-c37eaf3e6849","Type":"ContainerDied","Data":"22e8c3662bcf88f1d632ea88c087a2374f73b054a4a4068f75d3b00fe579aad8"}
Sep 16 19:06:50.741520 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:50.741483    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q" event={"ID":"4803396d-af3a-4d12-8fe2-c37eaf3e6849","Type":"ContainerStarted","Data":"db2a893fd4d02c8c9aef50be5edda73598544a76b727a449c0b516738d9e73e1"}
Sep 16 19:06:50.741927 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:50.741574    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q" event={"ID":"4803396d-af3a-4d12-8fe2-c37eaf3e6849","Type":"ContainerStarted","Data":"84fa105fa3e742b32ee43847ad6714e82c474018a32de5b5c494addc4488d167"}
Sep 16 19:06:50.741927 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:50.741868    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"
Sep 16 19:06:50.741927 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:50.741915    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"
Sep 16 19:06:50.763754 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:50.763429    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q" podStartSLOduration=4.055477369 podStartE2EDuration="32.763411174s" totalImagesPullingTime="0s" totalInitContainerRuntime="28.707933805s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:06:50.762490125 +0000 UTC m=+1320.838259836" watchObservedRunningTime="2026-09-16 19:06:50.763411174 +0000 UTC m=+1320.839180807"
Sep 16 19:06:55.208443 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:55.208412    2554 scope.go:117] "RemoveContainer" containerID="1796578f51addde3f088be39a3cff7a8af4ed468e591e21b4190338dcd4aad61"
Sep 16 19:06:55.216575 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:55.216551    2554 scope.go:117] "RemoveContainer" containerID="53e1e563bfddd449ae6107f4a58fb5f581112c65255e4db23e330add1777cefb"
Sep 16 19:06:55.224524 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:55.224504    2554 scope.go:117] "RemoveContainer" containerID="770d9cb7a5ba8e1084037dfc2900dd6d3d70aafece918d761e3c507783fdb6a4"
Sep 16 19:06:55.232807 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:55.232784    2554 scope.go:117] "RemoveContainer" containerID="1b4fabf3e8a4ee7e6bd8b5d2ab072c9390e3abb9402108d30e3d077c9267f907"
Sep 16 19:06:56.751445 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:06:56.751413    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"
Sep 16 19:07:42.008268 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:42.008236    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"]
Sep 16 19:07:42.010980 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:42.010962    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"
Sep 16 19:07:42.012938 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:42.012912    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Sep 16 19:07:42.013062 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:42.013034    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Sep 16 19:07:42.022400 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:42.022378    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"]
Sep 16 19:07:42.122452 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:42.122414    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg5bb\" (UniqueName: \"kubernetes.io/projected/9b55d3c4-1431-4be4-8d97-d395ec0e6712-kube-api-access-lg5bb\") pod \"isvc-sklearn-predictor-5d84548fd6-4d582\" (UID: \"9b55d3c4-1431-4be4-8d97-d395ec0e6712\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"
Sep 16 19:07:42.122452 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:42.122450    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b55d3c4-1431-4be4-8d97-d395ec0e6712-cabundle-cert\") pod \"isvc-sklearn-predictor-5d84548fd6-4d582\" (UID: \"9b55d3c4-1431-4be4-8d97-d395ec0e6712\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"
Sep 16 19:07:42.122705 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:42.122478    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b55d3c4-1431-4be4-8d97-d395ec0e6712-kserve-provision-location\") pod \"isvc-sklearn-predictor-5d84548fd6-4d582\" (UID: \"9b55d3c4-1431-4be4-8d97-d395ec0e6712\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"
Sep 16 19:07:42.122705 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:42.122615    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b55d3c4-1431-4be4-8d97-d395ec0e6712-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5d84548fd6-4d582\" (UID: \"9b55d3c4-1431-4be4-8d97-d395ec0e6712\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"
Sep 16 19:07:42.122705 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:42.122675    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b55d3c4-1431-4be4-8d97-d395ec0e6712-proxy-tls\") pod \"isvc-sklearn-predictor-5d84548fd6-4d582\" (UID: \"9b55d3c4-1431-4be4-8d97-d395ec0e6712\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"
Sep 16 19:07:42.223406 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:42.223358    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg5bb\" (UniqueName: \"kubernetes.io/projected/9b55d3c4-1431-4be4-8d97-d395ec0e6712-kube-api-access-lg5bb\") pod \"isvc-sklearn-predictor-5d84548fd6-4d582\" (UID: \"9b55d3c4-1431-4be4-8d97-d395ec0e6712\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"
Sep 16 19:07:42.223406 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:42.223408    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b55d3c4-1431-4be4-8d97-d395ec0e6712-cabundle-cert\") pod \"isvc-sklearn-predictor-5d84548fd6-4d582\" (UID: \"9b55d3c4-1431-4be4-8d97-d395ec0e6712\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"
Sep 16 19:07:42.223655 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:42.223438    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b55d3c4-1431-4be4-8d97-d395ec0e6712-kserve-provision-location\") pod \"isvc-sklearn-predictor-5d84548fd6-4d582\" (UID: \"9b55d3c4-1431-4be4-8d97-d395ec0e6712\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"
Sep 16 19:07:42.223655 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:42.223483    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b55d3c4-1431-4be4-8d97-d395ec0e6712-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5d84548fd6-4d582\" (UID: \"9b55d3c4-1431-4be4-8d97-d395ec0e6712\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"
Sep 16 19:07:42.223655 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:42.223532    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b55d3c4-1431-4be4-8d97-d395ec0e6712-proxy-tls\") pod \"isvc-sklearn-predictor-5d84548fd6-4d582\" (UID: \"9b55d3c4-1431-4be4-8d97-d395ec0e6712\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"
Sep 16 19:07:42.223655 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:07:42.223645    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-predictor-serving-cert: secret "isvc-sklearn-predictor-serving-cert" not found
Sep 16 19:07:42.224072 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:07:42.223708    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b55d3c4-1431-4be4-8d97-d395ec0e6712-proxy-tls podName:9b55d3c4-1431-4be4-8d97-d395ec0e6712 nodeName:}" failed. No retries permitted until 2026-09-16 19:07:42.723688586 +0000 UTC m=+1372.799458196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9b55d3c4-1431-4be4-8d97-d395ec0e6712-proxy-tls") pod "isvc-sklearn-predictor-5d84548fd6-4d582" (UID: "9b55d3c4-1431-4be4-8d97-d395ec0e6712") : secret "isvc-sklearn-predictor-serving-cert" not found
Sep 16 19:07:42.224072 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:42.223989    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b55d3c4-1431-4be4-8d97-d395ec0e6712-kserve-provision-location\") pod \"isvc-sklearn-predictor-5d84548fd6-4d582\" (UID: \"9b55d3c4-1431-4be4-8d97-d395ec0e6712\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"
Sep 16 19:07:42.224202 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:42.224176    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b55d3c4-1431-4be4-8d97-d395ec0e6712-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5d84548fd6-4d582\" (UID: \"9b55d3c4-1431-4be4-8d97-d395ec0e6712\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"
Sep 16 19:07:42.224292 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:42.224271    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b55d3c4-1431-4be4-8d97-d395ec0e6712-cabundle-cert\") pod \"isvc-sklearn-predictor-5d84548fd6-4d582\" (UID: \"9b55d3c4-1431-4be4-8d97-d395ec0e6712\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"
Sep 16 19:07:42.231904 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:42.231886    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg5bb\" (UniqueName: \"kubernetes.io/projected/9b55d3c4-1431-4be4-8d97-d395ec0e6712-kube-api-access-lg5bb\") pod \"isvc-sklearn-predictor-5d84548fd6-4d582\" (UID: \"9b55d3c4-1431-4be4-8d97-d395ec0e6712\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"
Sep 16 19:07:42.727078 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:42.727025    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b55d3c4-1431-4be4-8d97-d395ec0e6712-proxy-tls\") pod \"isvc-sklearn-predictor-5d84548fd6-4d582\" (UID: \"9b55d3c4-1431-4be4-8d97-d395ec0e6712\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"
Sep 16 19:07:42.729522 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:42.729503    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b55d3c4-1431-4be4-8d97-d395ec0e6712-proxy-tls\") pod \"isvc-sklearn-predictor-5d84548fd6-4d582\" (UID: \"9b55d3c4-1431-4be4-8d97-d395ec0e6712\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"
Sep 16 19:07:42.920895 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:42.920859    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"
Sep 16 19:07:43.191524 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:43.191500    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"]
Sep 16 19:07:43.193743 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:07:43.193704    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b55d3c4_1431_4be4_8d97_d395ec0e6712.slice/crio-122236be1ea51a89c4a3fd8827e770bfaa671b08dc135c89edc04034241bd12f WatchSource:0}: Error finding container 122236be1ea51a89c4a3fd8827e770bfaa671b08dc135c89edc04034241bd12f: Status 404 returned error can't find the container with id 122236be1ea51a89c4a3fd8827e770bfaa671b08dc135c89edc04034241bd12f
Sep 16 19:07:43.992855 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:43.992822    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582" event={"ID":"9b55d3c4-1431-4be4-8d97-d395ec0e6712","Type":"ContainerStarted","Data":"4fbac7575ed36d9a8d7caf04f1d4470729a2a499e931f731db0eda8c4ec946c9"}
Sep 16 19:07:43.993080 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:43.992878    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582" event={"ID":"9b55d3c4-1431-4be4-8d97-d395ec0e6712","Type":"ContainerStarted","Data":"122236be1ea51a89c4a3fd8827e770bfaa671b08dc135c89edc04034241bd12f"}
Sep 16 19:07:48.013761 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:48.013711    2554 generic.go:356] "Generic (PLEG): container finished" podID="9b55d3c4-1431-4be4-8d97-d395ec0e6712" containerID="4fbac7575ed36d9a8d7caf04f1d4470729a2a499e931f731db0eda8c4ec946c9" exitCode=0
Sep 16 19:07:48.014176 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:48.013785    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582" event={"ID":"9b55d3c4-1431-4be4-8d97-d395ec0e6712","Type":"ContainerDied","Data":"4fbac7575ed36d9a8d7caf04f1d4470729a2a499e931f731db0eda8c4ec946c9"}
Sep 16 19:07:49.020899 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:49.020866    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582" event={"ID":"9b55d3c4-1431-4be4-8d97-d395ec0e6712","Type":"ContainerStarted","Data":"df2daefb3a76e220cc2f8d2359b9a50d45bf0ea8b69f97731615c3f01396937c"}
Sep 16 19:07:49.021280 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:49.020925    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582" event={"ID":"9b55d3c4-1431-4be4-8d97-d395ec0e6712","Type":"ContainerStarted","Data":"4cea3b7449941aebfb4c180dddc51dee76c29c6ef4ceccbdc7162f663257bb24"}
Sep 16 19:07:49.021280 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:49.021245    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"
Sep 16 19:07:49.021372 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:49.021361    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"
Sep 16 19:07:49.022613 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:49.022587    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582" podUID="9b55d3c4-1431-4be4-8d97-d395ec0e6712" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:07:49.047134 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:49.047079    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582" podStartSLOduration=4.201463955 podStartE2EDuration="8.047064376s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.845600421s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:07:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:07:49.046121815 +0000 UTC m=+1379.121891450" watchObservedRunningTime="2026-09-16 19:07:49.047064376 +0000 UTC m=+1379.122834008"
Sep 16 19:07:50.026006 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:50.025973    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582" podUID="9b55d3c4-1431-4be4-8d97-d395ec0e6712" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:07:55.030124 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:55.030089    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"
Sep 16 19:07:55.030595 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:07:55.030568    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582" podUID="9b55d3c4-1431-4be4-8d97-d395ec0e6712" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:08:05.030505 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:05.030458    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582" podUID="9b55d3c4-1431-4be4-8d97-d395ec0e6712" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:08:15.031385 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:15.031343    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582" podUID="9b55d3c4-1431-4be4-8d97-d395ec0e6712" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:08:22.450028 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:22.450001    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"
Sep 16 19:08:25.031076 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:25.031035    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582" podUID="9b55d3c4-1431-4be4-8d97-d395ec0e6712" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:08:33.478904 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:33.478849    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q" podUID="4803396d-af3a-4d12-8fe2-c37eaf3e6849" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.39:8080/v2/models/isvc-autogluon-v2-variants/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:08:35.031361 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:35.031314    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582" podUID="9b55d3c4-1431-4be4-8d97-d395ec0e6712" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:08:35.548695 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:35.548666    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"]
Sep 16 19:08:35.548965 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:35.548939    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q" podUID="4803396d-af3a-4d12-8fe2-c37eaf3e6849" containerName="kserve-container" containerID="cri-o://84fa105fa3e742b32ee43847ad6714e82c474018a32de5b5c494addc4488d167" gracePeriod=30
Sep 16 19:08:35.549067 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:35.548989    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q" podUID="4803396d-af3a-4d12-8fe2-c37eaf3e6849" containerName="kube-rbac-proxy" containerID="cri-o://db2a893fd4d02c8c9aef50be5edda73598544a76b727a449c0b516738d9e73e1" gracePeriod=30
Sep 16 19:08:36.247734 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:36.247685    2554 generic.go:356] "Generic (PLEG): container finished" podID="4803396d-af3a-4d12-8fe2-c37eaf3e6849" containerID="db2a893fd4d02c8c9aef50be5edda73598544a76b727a449c0b516738d9e73e1" exitCode=2
Sep 16 19:08:36.247734 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:36.247736    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q" event={"ID":"4803396d-af3a-4d12-8fe2-c37eaf3e6849","Type":"ContainerDied","Data":"db2a893fd4d02c8c9aef50be5edda73598544a76b727a449c0b516738d9e73e1"}
Sep 16 19:08:36.746815 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:36.746771    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q" podUID="4803396d-af3a-4d12-8fe2-c37eaf3e6849" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Sep 16 19:08:36.992151 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:36.992125    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"
Sep 16 19:08:37.111242 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.111204    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4803396d-af3a-4d12-8fe2-c37eaf3e6849-proxy-tls\") pod \"4803396d-af3a-4d12-8fe2-c37eaf3e6849\" (UID: \"4803396d-af3a-4d12-8fe2-c37eaf3e6849\") "
Sep 16 19:08:37.111437 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.111260    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4803396d-af3a-4d12-8fe2-c37eaf3e6849-kube-rbac-proxy-sar-config\") pod \"4803396d-af3a-4d12-8fe2-c37eaf3e6849\" (UID: \"4803396d-af3a-4d12-8fe2-c37eaf3e6849\") "
Sep 16 19:08:37.111437 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.111281    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4803396d-af3a-4d12-8fe2-c37eaf3e6849-cabundle-cert\") pod \"4803396d-af3a-4d12-8fe2-c37eaf3e6849\" (UID: \"4803396d-af3a-4d12-8fe2-c37eaf3e6849\") "
Sep 16 19:08:37.111548 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.111439    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cczz8\" (UniqueName: \"kubernetes.io/projected/4803396d-af3a-4d12-8fe2-c37eaf3e6849-kube-api-access-cczz8\") pod \"4803396d-af3a-4d12-8fe2-c37eaf3e6849\" (UID: \"4803396d-af3a-4d12-8fe2-c37eaf3e6849\") "
Sep 16 19:08:37.111609 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.111548    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4803396d-af3a-4d12-8fe2-c37eaf3e6849-kserve-provision-location\") pod \"4803396d-af3a-4d12-8fe2-c37eaf3e6849\" (UID: \"4803396d-af3a-4d12-8fe2-c37eaf3e6849\") "
Sep 16 19:08:37.111668 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.111641    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4803396d-af3a-4d12-8fe2-c37eaf3e6849-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4803396d-af3a-4d12-8fe2-c37eaf3e6849" (UID: "4803396d-af3a-4d12-8fe2-c37eaf3e6849"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:37.111747 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.111654    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4803396d-af3a-4d12-8fe2-c37eaf3e6849-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4803396d-af3a-4d12-8fe2-c37eaf3e6849" (UID: "4803396d-af3a-4d12-8fe2-c37eaf3e6849"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:08:37.112029 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.112008    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4803396d-af3a-4d12-8fe2-c37eaf3e6849-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:08:37.112109 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.112036    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4803396d-af3a-4d12-8fe2-c37eaf3e6849-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:08:37.113492 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.113466    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4803396d-af3a-4d12-8fe2-c37eaf3e6849-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4803396d-af3a-4d12-8fe2-c37eaf3e6849" (UID: "4803396d-af3a-4d12-8fe2-c37eaf3e6849"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:08:37.113564 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.113526    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4803396d-af3a-4d12-8fe2-c37eaf3e6849-kube-api-access-cczz8" (OuterVolumeSpecName: "kube-api-access-cczz8") pod "4803396d-af3a-4d12-8fe2-c37eaf3e6849" (UID: "4803396d-af3a-4d12-8fe2-c37eaf3e6849"). InnerVolumeSpecName "kube-api-access-cczz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:08:37.116264 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.116241    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4803396d-af3a-4d12-8fe2-c37eaf3e6849-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4803396d-af3a-4d12-8fe2-c37eaf3e6849" (UID: "4803396d-af3a-4d12-8fe2-c37eaf3e6849"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:08:37.213417 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.213383    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cczz8\" (UniqueName: \"kubernetes.io/projected/4803396d-af3a-4d12-8fe2-c37eaf3e6849-kube-api-access-cczz8\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:08:37.213417 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.213412    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4803396d-af3a-4d12-8fe2-c37eaf3e6849-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:08:37.213417 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.213421    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4803396d-af3a-4d12-8fe2-c37eaf3e6849-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:08:37.255177 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.255140    2554 generic.go:356] "Generic (PLEG): container finished" podID="4803396d-af3a-4d12-8fe2-c37eaf3e6849" containerID="84fa105fa3e742b32ee43847ad6714e82c474018a32de5b5c494addc4488d167" exitCode=0
Sep 16 19:08:37.255567 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.255222    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"
Sep 16 19:08:37.255567 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.255225    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q" event={"ID":"4803396d-af3a-4d12-8fe2-c37eaf3e6849","Type":"ContainerDied","Data":"84fa105fa3e742b32ee43847ad6714e82c474018a32de5b5c494addc4488d167"}
Sep 16 19:08:37.255567 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.255358    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q" event={"ID":"4803396d-af3a-4d12-8fe2-c37eaf3e6849","Type":"ContainerDied","Data":"d5ffbb9e42a59a93ffb5cc4ef24326cdac3c616ed64176a4260516035aa81e3d"}
Sep 16 19:08:37.255567 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.255397    2554 scope.go:117] "RemoveContainer" containerID="db2a893fd4d02c8c9aef50be5edda73598544a76b727a449c0b516738d9e73e1"
Sep 16 19:08:37.264633 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.264617    2554 scope.go:117] "RemoveContainer" containerID="84fa105fa3e742b32ee43847ad6714e82c474018a32de5b5c494addc4488d167"
Sep 16 19:08:37.272179 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.272159    2554 scope.go:117] "RemoveContainer" containerID="22e8c3662bcf88f1d632ea88c087a2374f73b054a4a4068f75d3b00fe579aad8"
Sep 16 19:08:37.277788 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.277766    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"]
Sep 16 19:08:37.280919 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.280900    2554 scope.go:117] "RemoveContainer" containerID="db2a893fd4d02c8c9aef50be5edda73598544a76b727a449c0b516738d9e73e1"
Sep 16 19:08:37.281193 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:08:37.281174    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db2a893fd4d02c8c9aef50be5edda73598544a76b727a449c0b516738d9e73e1\": container with ID starting with db2a893fd4d02c8c9aef50be5edda73598544a76b727a449c0b516738d9e73e1 not found: ID does not exist" containerID="db2a893fd4d02c8c9aef50be5edda73598544a76b727a449c0b516738d9e73e1"
Sep 16 19:08:37.281280 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.281205    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db2a893fd4d02c8c9aef50be5edda73598544a76b727a449c0b516738d9e73e1"} err="failed to get container status \"db2a893fd4d02c8c9aef50be5edda73598544a76b727a449c0b516738d9e73e1\": rpc error: code = NotFound desc = could not find container \"db2a893fd4d02c8c9aef50be5edda73598544a76b727a449c0b516738d9e73e1\": container with ID starting with db2a893fd4d02c8c9aef50be5edda73598544a76b727a449c0b516738d9e73e1 not found: ID does not exist"
Sep 16 19:08:37.281280 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.281231    2554 scope.go:117] "RemoveContainer" containerID="84fa105fa3e742b32ee43847ad6714e82c474018a32de5b5c494addc4488d167"
Sep 16 19:08:37.281489 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.281471    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-nwt7q"]
Sep 16 19:08:37.281564 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:08:37.281507    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84fa105fa3e742b32ee43847ad6714e82c474018a32de5b5c494addc4488d167\": container with ID starting with 84fa105fa3e742b32ee43847ad6714e82c474018a32de5b5c494addc4488d167 not found: ID does not exist" containerID="84fa105fa3e742b32ee43847ad6714e82c474018a32de5b5c494addc4488d167"
Sep 16 19:08:37.281564 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.281533    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84fa105fa3e742b32ee43847ad6714e82c474018a32de5b5c494addc4488d167"} err="failed to get container status \"84fa105fa3e742b32ee43847ad6714e82c474018a32de5b5c494addc4488d167\": rpc error: code = NotFound desc = could not find container \"84fa105fa3e742b32ee43847ad6714e82c474018a32de5b5c494addc4488d167\": container with ID starting with 84fa105fa3e742b32ee43847ad6714e82c474018a32de5b5c494addc4488d167 not found: ID does not exist"
Sep 16 19:08:37.281564 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.281555    2554 scope.go:117] "RemoveContainer" containerID="22e8c3662bcf88f1d632ea88c087a2374f73b054a4a4068f75d3b00fe579aad8"
Sep 16 19:08:37.281868 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:08:37.281851    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22e8c3662bcf88f1d632ea88c087a2374f73b054a4a4068f75d3b00fe579aad8\": container with ID starting with 22e8c3662bcf88f1d632ea88c087a2374f73b054a4a4068f75d3b00fe579aad8 not found: ID does not exist" containerID="22e8c3662bcf88f1d632ea88c087a2374f73b054a4a4068f75d3b00fe579aad8"
Sep 16 19:08:37.281922 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:37.281872    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22e8c3662bcf88f1d632ea88c087a2374f73b054a4a4068f75d3b00fe579aad8"} err="failed to get container status \"22e8c3662bcf88f1d632ea88c087a2374f73b054a4a4068f75d3b00fe579aad8\": rpc error: code = NotFound desc = could not find container \"22e8c3662bcf88f1d632ea88c087a2374f73b054a4a4068f75d3b00fe579aad8\": container with ID starting with 22e8c3662bcf88f1d632ea88c087a2374f73b054a4a4068f75d3b00fe579aad8 not found: ID does not exist"
Sep 16 19:08:38.443080 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:38.443050    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4803396d-af3a-4d12-8fe2-c37eaf3e6849" path="/var/lib/kubelet/pods/4803396d-af3a-4d12-8fe2-c37eaf3e6849/volumes"
Sep 16 19:08:45.030796 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:45.030701    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582" podUID="9b55d3c4-1431-4be4-8d97-d395ec0e6712" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:08:55.031311 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:08:55.031274    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582" podUID="9b55d3c4-1431-4be4-8d97-d395ec0e6712" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:09:05.031726 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:05.031697    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"
Sep 16 19:09:12.087913 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:12.087879    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"]
Sep 16 19:09:12.088400 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:12.088147    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582" podUID="9b55d3c4-1431-4be4-8d97-d395ec0e6712" containerName="kserve-container" containerID="cri-o://4cea3b7449941aebfb4c180dddc51dee76c29c6ef4ceccbdc7162f663257bb24" gracePeriod=30
Sep 16 19:09:12.088400 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:12.088190    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582" podUID="9b55d3c4-1431-4be4-8d97-d395ec0e6712" containerName="kube-rbac-proxy" containerID="cri-o://df2daefb3a76e220cc2f8d2359b9a50d45bf0ea8b69f97731615c3f01396937c" gracePeriod=30
Sep 16 19:09:12.410221 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:12.410142    2554 generic.go:356] "Generic (PLEG): container finished" podID="9b55d3c4-1431-4be4-8d97-d395ec0e6712" containerID="df2daefb3a76e220cc2f8d2359b9a50d45bf0ea8b69f97731615c3f01396937c" exitCode=2
Sep 16 19:09:12.410358 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:12.410216    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582" event={"ID":"9b55d3c4-1431-4be4-8d97-d395ec0e6712","Type":"ContainerDied","Data":"df2daefb3a76e220cc2f8d2359b9a50d45bf0ea8b69f97731615c3f01396937c"}
Sep 16 19:09:15.027221 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:15.027171    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582" podUID="9b55d3c4-1431-4be4-8d97-d395ec0e6712" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Sep 16 19:09:15.030485 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:15.030461    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582" podUID="9b55d3c4-1431-4be4-8d97-d395ec0e6712" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Sep 16 19:09:15.929088 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:15.929065    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"
Sep 16 19:09:16.006955 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.006898    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b55d3c4-1431-4be4-8d97-d395ec0e6712-proxy-tls\") pod \"9b55d3c4-1431-4be4-8d97-d395ec0e6712\" (UID: \"9b55d3c4-1431-4be4-8d97-d395ec0e6712\") "
Sep 16 19:09:16.007061 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.006981    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b55d3c4-1431-4be4-8d97-d395ec0e6712-kserve-provision-location\") pod \"9b55d3c4-1431-4be4-8d97-d395ec0e6712\" (UID: \"9b55d3c4-1431-4be4-8d97-d395ec0e6712\") "
Sep 16 19:09:16.007061 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.007006    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b55d3c4-1431-4be4-8d97-d395ec0e6712-cabundle-cert\") pod \"9b55d3c4-1431-4be4-8d97-d395ec0e6712\" (UID: \"9b55d3c4-1431-4be4-8d97-d395ec0e6712\") "
Sep 16 19:09:16.007061 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.007029    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lg5bb\" (UniqueName: \"kubernetes.io/projected/9b55d3c4-1431-4be4-8d97-d395ec0e6712-kube-api-access-lg5bb\") pod \"9b55d3c4-1431-4be4-8d97-d395ec0e6712\" (UID: \"9b55d3c4-1431-4be4-8d97-d395ec0e6712\") "
Sep 16 19:09:16.007061 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.007049    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b55d3c4-1431-4be4-8d97-d395ec0e6712-kube-rbac-proxy-sar-config\") pod \"9b55d3c4-1431-4be4-8d97-d395ec0e6712\" (UID: \"9b55d3c4-1431-4be4-8d97-d395ec0e6712\") "
Sep 16 19:09:16.007399 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.007310    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b55d3c4-1431-4be4-8d97-d395ec0e6712-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9b55d3c4-1431-4be4-8d97-d395ec0e6712" (UID: "9b55d3c4-1431-4be4-8d97-d395ec0e6712"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:09:16.007399 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.007381    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b55d3c4-1431-4be4-8d97-d395ec0e6712-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9b55d3c4-1431-4be4-8d97-d395ec0e6712" (UID: "9b55d3c4-1431-4be4-8d97-d395ec0e6712"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:16.007506 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.007435    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b55d3c4-1431-4be4-8d97-d395ec0e6712-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9b55d3c4-1431-4be4-8d97-d395ec0e6712" (UID: "9b55d3c4-1431-4be4-8d97-d395ec0e6712"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:09:16.008930 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.008906    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b55d3c4-1431-4be4-8d97-d395ec0e6712-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9b55d3c4-1431-4be4-8d97-d395ec0e6712" (UID: "9b55d3c4-1431-4be4-8d97-d395ec0e6712"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:09:16.009016 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.008985    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b55d3c4-1431-4be4-8d97-d395ec0e6712-kube-api-access-lg5bb" (OuterVolumeSpecName: "kube-api-access-lg5bb") pod "9b55d3c4-1431-4be4-8d97-d395ec0e6712" (UID: "9b55d3c4-1431-4be4-8d97-d395ec0e6712"). InnerVolumeSpecName "kube-api-access-lg5bb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:09:16.108466 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.108446    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b55d3c4-1431-4be4-8d97-d395ec0e6712-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:09:16.108466 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.108466    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b55d3c4-1431-4be4-8d97-d395ec0e6712-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:09:16.108817 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.108476    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg5bb\" (UniqueName: \"kubernetes.io/projected/9b55d3c4-1431-4be4-8d97-d395ec0e6712-kube-api-access-lg5bb\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:09:16.108817 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.108486    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b55d3c4-1431-4be4-8d97-d395ec0e6712-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:09:16.108817 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.108495    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b55d3c4-1431-4be4-8d97-d395ec0e6712-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:09:16.439286 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.439221    2554 generic.go:356] "Generic (PLEG): container finished" podID="9b55d3c4-1431-4be4-8d97-d395ec0e6712" containerID="4cea3b7449941aebfb4c180dddc51dee76c29c6ef4ceccbdc7162f663257bb24" exitCode=0
Sep 16 19:09:16.439396 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.439305    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"
Sep 16 19:09:16.443219 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.443163    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582" event={"ID":"9b55d3c4-1431-4be4-8d97-d395ec0e6712","Type":"ContainerDied","Data":"4cea3b7449941aebfb4c180dddc51dee76c29c6ef4ceccbdc7162f663257bb24"}
Sep 16 19:09:16.443219 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.443211    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582" event={"ID":"9b55d3c4-1431-4be4-8d97-d395ec0e6712","Type":"ContainerDied","Data":"122236be1ea51a89c4a3fd8827e770bfaa671b08dc135c89edc04034241bd12f"}
Sep 16 19:09:16.443416 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.443244    2554 scope.go:117] "RemoveContainer" containerID="df2daefb3a76e220cc2f8d2359b9a50d45bf0ea8b69f97731615c3f01396937c"
Sep 16 19:09:16.451883 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.451866    2554 scope.go:117] "RemoveContainer" containerID="4cea3b7449941aebfb4c180dddc51dee76c29c6ef4ceccbdc7162f663257bb24"
Sep 16 19:09:16.459471 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.459452    2554 scope.go:117] "RemoveContainer" containerID="4fbac7575ed36d9a8d7caf04f1d4470729a2a499e931f731db0eda8c4ec946c9"
Sep 16 19:09:16.464234 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.464210    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"]
Sep 16 19:09:16.467329 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.467314    2554 scope.go:117] "RemoveContainer" containerID="df2daefb3a76e220cc2f8d2359b9a50d45bf0ea8b69f97731615c3f01396937c"
Sep 16 19:09:16.467572 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:09:16.467552    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df2daefb3a76e220cc2f8d2359b9a50d45bf0ea8b69f97731615c3f01396937c\": container with ID starting with df2daefb3a76e220cc2f8d2359b9a50d45bf0ea8b69f97731615c3f01396937c not found: ID does not exist" containerID="df2daefb3a76e220cc2f8d2359b9a50d45bf0ea8b69f97731615c3f01396937c"
Sep 16 19:09:16.467645 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.467580    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df2daefb3a76e220cc2f8d2359b9a50d45bf0ea8b69f97731615c3f01396937c"} err="failed to get container status \"df2daefb3a76e220cc2f8d2359b9a50d45bf0ea8b69f97731615c3f01396937c\": rpc error: code = NotFound desc = could not find container \"df2daefb3a76e220cc2f8d2359b9a50d45bf0ea8b69f97731615c3f01396937c\": container with ID starting with df2daefb3a76e220cc2f8d2359b9a50d45bf0ea8b69f97731615c3f01396937c not found: ID does not exist"
Sep 16 19:09:16.467645 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.467598    2554 scope.go:117] "RemoveContainer" containerID="4cea3b7449941aebfb4c180dddc51dee76c29c6ef4ceccbdc7162f663257bb24"
Sep 16 19:09:16.467843 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:09:16.467825    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cea3b7449941aebfb4c180dddc51dee76c29c6ef4ceccbdc7162f663257bb24\": container with ID starting with 4cea3b7449941aebfb4c180dddc51dee76c29c6ef4ceccbdc7162f663257bb24 not found: ID does not exist" containerID="4cea3b7449941aebfb4c180dddc51dee76c29c6ef4ceccbdc7162f663257bb24"
Sep 16 19:09:16.467905 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.467848    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cea3b7449941aebfb4c180dddc51dee76c29c6ef4ceccbdc7162f663257bb24"} err="failed to get container status \"4cea3b7449941aebfb4c180dddc51dee76c29c6ef4ceccbdc7162f663257bb24\": rpc error: code = NotFound desc = could not find container \"4cea3b7449941aebfb4c180dddc51dee76c29c6ef4ceccbdc7162f663257bb24\": container with ID starting with 4cea3b7449941aebfb4c180dddc51dee76c29c6ef4ceccbdc7162f663257bb24 not found: ID does not exist"
Sep 16 19:09:16.467905 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.467863    2554 scope.go:117] "RemoveContainer" containerID="4fbac7575ed36d9a8d7caf04f1d4470729a2a499e931f731db0eda8c4ec946c9"
Sep 16 19:09:16.468108 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:09:16.468092    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fbac7575ed36d9a8d7caf04f1d4470729a2a499e931f731db0eda8c4ec946c9\": container with ID starting with 4fbac7575ed36d9a8d7caf04f1d4470729a2a499e931f731db0eda8c4ec946c9 not found: ID does not exist" containerID="4fbac7575ed36d9a8d7caf04f1d4470729a2a499e931f731db0eda8c4ec946c9"
Sep 16 19:09:16.468165 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.468112    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fbac7575ed36d9a8d7caf04f1d4470729a2a499e931f731db0eda8c4ec946c9"} err="failed to get container status \"4fbac7575ed36d9a8d7caf04f1d4470729a2a499e931f731db0eda8c4ec946c9\": rpc error: code = NotFound desc = could not find container \"4fbac7575ed36d9a8d7caf04f1d4470729a2a499e931f731db0eda8c4ec946c9\": container with ID starting with 4fbac7575ed36d9a8d7caf04f1d4470729a2a499e931f731db0eda8c4ec946c9 not found: ID does not exist"
Sep 16 19:09:16.468222 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:16.468190    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5d84548fd6-4d582"]
Sep 16 19:09:17.343023 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.342993    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"]
Sep 16 19:09:17.343379 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.343345    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4803396d-af3a-4d12-8fe2-c37eaf3e6849" containerName="kube-rbac-proxy"
Sep 16 19:09:17.343379 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.343357    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4803396d-af3a-4d12-8fe2-c37eaf3e6849" containerName="kube-rbac-proxy"
Sep 16 19:09:17.343379 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.343367    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b55d3c4-1431-4be4-8d97-d395ec0e6712" containerName="kserve-container"
Sep 16 19:09:17.343379 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.343374    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b55d3c4-1431-4be4-8d97-d395ec0e6712" containerName="kserve-container"
Sep 16 19:09:17.343508 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.343388    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b55d3c4-1431-4be4-8d97-d395ec0e6712" containerName="storage-initializer"
Sep 16 19:09:17.343508 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.343396    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b55d3c4-1431-4be4-8d97-d395ec0e6712" containerName="storage-initializer"
Sep 16 19:09:17.343508 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.343403    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b55d3c4-1431-4be4-8d97-d395ec0e6712" containerName="kube-rbac-proxy"
Sep 16 19:09:17.343508 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.343409    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b55d3c4-1431-4be4-8d97-d395ec0e6712" containerName="kube-rbac-proxy"
Sep 16 19:09:17.343508 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.343424    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4803396d-af3a-4d12-8fe2-c37eaf3e6849" containerName="storage-initializer"
Sep 16 19:09:17.343508 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.343429    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4803396d-af3a-4d12-8fe2-c37eaf3e6849" containerName="storage-initializer"
Sep 16 19:09:17.343508 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.343447    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4803396d-af3a-4d12-8fe2-c37eaf3e6849" containerName="kserve-container"
Sep 16 19:09:17.343508 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.343452    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4803396d-af3a-4d12-8fe2-c37eaf3e6849" containerName="kserve-container"
Sep 16 19:09:17.343508 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.343499    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4803396d-af3a-4d12-8fe2-c37eaf3e6849" containerName="kube-rbac-proxy"
Sep 16 19:09:17.343508 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.343507    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4803396d-af3a-4d12-8fe2-c37eaf3e6849" containerName="kserve-container"
Sep 16 19:09:17.343839 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.343516    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b55d3c4-1431-4be4-8d97-d395ec0e6712" containerName="kserve-container"
Sep 16 19:09:17.343839 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.343524    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b55d3c4-1431-4be4-8d97-d395ec0e6712" containerName="kube-rbac-proxy"
Sep 16 19:09:17.348335 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.348316    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"
Sep 16 19:09:17.351734 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.351697    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:09:17.351734 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.351712    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 16 19:09:17.352033 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.352012    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:09:17.352101 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.352066    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Sep 16 19:09:17.352211 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.352193    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8gklb\""
Sep 16 19:09:17.352482 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.352460    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:09:17.352638 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.352522    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:09:17.358142 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.358121    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"]
Sep 16 19:09:17.419128 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.419103    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7z676\" (UID: \"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"
Sep 16 19:09:17.419275 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.419138    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7z676\" (UID: \"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"
Sep 16 19:09:17.419275 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.419221    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7z676\" (UID: \"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"
Sep 16 19:09:17.419394 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.419295    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6md9t\" (UniqueName: \"kubernetes.io/projected/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-kube-api-access-6md9t\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7z676\" (UID: \"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"
Sep 16 19:09:17.419394 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.419323    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7z676\" (UID: \"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"
Sep 16 19:09:17.519885 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.519863    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6md9t\" (UniqueName: \"kubernetes.io/projected/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-kube-api-access-6md9t\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7z676\" (UID: \"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"
Sep 16 19:09:17.520005 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.519890    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7z676\" (UID: \"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"
Sep 16 19:09:17.520005 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.519937    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7z676\" (UID: \"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"
Sep 16 19:09:17.520139 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:09:17.520013    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-serving-cert: secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Sep 16 19:09:17.520139 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.520013    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7z676\" (UID: \"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"
Sep 16 19:09:17.520139 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:09:17.520056    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-proxy-tls podName:746b0f1d-2fa8-4aaa-bf38-f3f50e631c34 nodeName:}" failed. No retries permitted until 2026-09-16 19:09:18.020042607 +0000 UTC m=+1468.095812216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-proxy-tls") pod "sklearn-v2-mlserver-predictor-7f568cdc6b-7z676" (UID: "746b0f1d-2fa8-4aaa-bf38-f3f50e631c34") : secret "sklearn-v2-mlserver-predictor-serving-cert" not found
Sep 16 19:09:17.520139 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.520089    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7z676\" (UID: \"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"
Sep 16 19:09:17.520326 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.520256    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7z676\" (UID: \"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"
Sep 16 19:09:17.520604 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.520586    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7z676\" (UID: \"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"
Sep 16 19:09:17.520698 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.520682    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7z676\" (UID: \"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"
Sep 16 19:09:17.528488 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:17.528461    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6md9t\" (UniqueName: \"kubernetes.io/projected/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-kube-api-access-6md9t\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7z676\" (UID: \"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"
Sep 16 19:09:18.025058 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:18.025022    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7z676\" (UID: \"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"
Sep 16 19:09:18.027339 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:18.027308    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-7z676\" (UID: \"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"
Sep 16 19:09:18.260148 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:18.260122    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"
Sep 16 19:09:18.443747 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:18.443702    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b55d3c4-1431-4be4-8d97-d395ec0e6712" path="/var/lib/kubelet/pods/9b55d3c4-1431-4be4-8d97-d395ec0e6712/volumes"
Sep 16 19:09:18.535224 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:18.535202    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"]
Sep 16 19:09:18.536622 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:09:18.536592    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746b0f1d_2fa8_4aaa_bf38_f3f50e631c34.slice/crio-55fbe50a80645c6560ddaaf0328da2e02ce0cf162e5a9aa1ef80ae47069be16d WatchSource:0}: Error finding container 55fbe50a80645c6560ddaaf0328da2e02ce0cf162e5a9aa1ef80ae47069be16d: Status 404 returned error can't find the container with id 55fbe50a80645c6560ddaaf0328da2e02ce0cf162e5a9aa1ef80ae47069be16d
Sep 16 19:09:19.455265 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:19.455232    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676" event={"ID":"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34","Type":"ContainerStarted","Data":"d585c9b5d5b1a85c64f8f7143ee5ea4ea00a090772c7f47c29658868012077e0"}
Sep 16 19:09:19.455654 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:19.455287    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676" event={"ID":"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34","Type":"ContainerStarted","Data":"55fbe50a80645c6560ddaaf0328da2e02ce0cf162e5a9aa1ef80ae47069be16d"}
Sep 16 19:09:23.474492 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:23.474458    2554 generic.go:356] "Generic (PLEG): container finished" podID="746b0f1d-2fa8-4aaa-bf38-f3f50e631c34" containerID="d585c9b5d5b1a85c64f8f7143ee5ea4ea00a090772c7f47c29658868012077e0" exitCode=0
Sep 16 19:09:23.474895 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:23.474528    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676" event={"ID":"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34","Type":"ContainerDied","Data":"d585c9b5d5b1a85c64f8f7143ee5ea4ea00a090772c7f47c29658868012077e0"}
Sep 16 19:09:23.475618 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:23.475601    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:09:50.448917 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:50.448886    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/2.log"
Sep 16 19:09:50.451323 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:50.451260    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/2.log"
Sep 16 19:09:50.458667 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:50.458646    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovn-acl-logging/0.log"
Sep 16 19:09:50.460973 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:09:50.460956    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovn-acl-logging/0.log"
Sep 16 19:11:31.189447 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:11:31.189349    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676" event={"ID":"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34","Type":"ContainerStarted","Data":"a6178eff833b8ea9cdda1a891b1f37c9e104aaa5c2ec9a3e05257c3fcc18f901"}
Sep 16 19:11:31.189867 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:11:31.189453    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"
Sep 16 19:11:31.189867 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:11:31.189476    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"
Sep 16 19:11:31.189867 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:11:31.189493    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676" event={"ID":"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34","Type":"ContainerStarted","Data":"3ee9395a8c080e08bd222ba9e65e5ed505aa408a35ec4f44f4c3d63be4801600"}
Sep 16 19:11:31.212221 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:11:31.212163    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676" podStartSLOduration=3.33081095 podStartE2EDuration="2m14.212147955s" totalImagesPullingTime="2m6.981586367s" totalInitContainerRuntime="3.899750638s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:09:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:11:31.210320361 +0000 UTC m=+1601.286090006" watchObservedRunningTime="2026-09-16 19:11:31.212147955 +0000 UTC m=+1601.287917587"
Sep 16 19:11:37.199472 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:11:37.199438    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"
Sep 16 19:12:07.451954 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:07.451866    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676" podUID="746b0f1d-2fa8-4aaa-bf38-f3f50e631c34" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 16 19:12:17.202268 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:17.202232    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"
Sep 16 19:12:27.509000 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:27.508967    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"]
Sep 16 19:12:27.509459 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:27.509271    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676" podUID="746b0f1d-2fa8-4aaa-bf38-f3f50e631c34" containerName="kserve-container" containerID="cri-o://3ee9395a8c080e08bd222ba9e65e5ed505aa408a35ec4f44f4c3d63be4801600" gracePeriod=30
Sep 16 19:12:27.509459 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:27.509330    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676" podUID="746b0f1d-2fa8-4aaa-bf38-f3f50e631c34" containerName="kube-rbac-proxy" containerID="cri-o://a6178eff833b8ea9cdda1a891b1f37c9e104aaa5c2ec9a3e05257c3fcc18f901" gracePeriod=30
Sep 16 19:12:28.467687 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:28.467652    2554 generic.go:356] "Generic (PLEG): container finished" podID="746b0f1d-2fa8-4aaa-bf38-f3f50e631c34" containerID="a6178eff833b8ea9cdda1a891b1f37c9e104aaa5c2ec9a3e05257c3fcc18f901" exitCode=2
Sep 16 19:12:28.467894 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:28.467733    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676" event={"ID":"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34","Type":"ContainerDied","Data":"a6178eff833b8ea9cdda1a891b1f37c9e104aaa5c2ec9a3e05257c3fcc18f901"}
Sep 16 19:12:32.195368 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:32.195323    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676" podUID="746b0f1d-2fa8-4aaa-bf38-f3f50e631c34" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Sep 16 19:12:32.708276 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:32.708248    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p"]
Sep 16 19:12:32.711090 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:32.711071    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p"
Sep 16 19:12:32.713194 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:32.713171    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Sep 16 19:12:32.713294 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:32.713174    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:12:32.727746 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:32.727707    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p"]
Sep 16 19:12:32.785145 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:32.785121    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1874e8b8-c6af-4b88-ba1f-566e2f755881-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-759986c4-v7w8p\" (UID: \"1874e8b8-c6af-4b88-ba1f-566e2f755881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p"
Sep 16 19:12:32.785264 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:32.785152    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdfsg\" (UniqueName: \"kubernetes.io/projected/1874e8b8-c6af-4b88-ba1f-566e2f755881-kube-api-access-sdfsg\") pod \"isvc-sklearn-runtime-predictor-759986c4-v7w8p\" (UID: \"1874e8b8-c6af-4b88-ba1f-566e2f755881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p"
Sep 16 19:12:32.785264 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:32.785178    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1874e8b8-c6af-4b88-ba1f-566e2f755881-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-759986c4-v7w8p\" (UID: \"1874e8b8-c6af-4b88-ba1f-566e2f755881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p"
Sep 16 19:12:32.785264 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:32.785241    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1874e8b8-c6af-4b88-ba1f-566e2f755881-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-759986c4-v7w8p\" (UID: \"1874e8b8-c6af-4b88-ba1f-566e2f755881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p"
Sep 16 19:12:32.785391 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:32.785273    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1874e8b8-c6af-4b88-ba1f-566e2f755881-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-759986c4-v7w8p\" (UID: \"1874e8b8-c6af-4b88-ba1f-566e2f755881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p"
Sep 16 19:12:32.886453 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:32.886420    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1874e8b8-c6af-4b88-ba1f-566e2f755881-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-759986c4-v7w8p\" (UID: \"1874e8b8-c6af-4b88-ba1f-566e2f755881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p"
Sep 16 19:12:32.886582 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:32.886483    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1874e8b8-c6af-4b88-ba1f-566e2f755881-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-759986c4-v7w8p\" (UID: \"1874e8b8-c6af-4b88-ba1f-566e2f755881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p"
Sep 16 19:12:32.886582 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:32.886520    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1874e8b8-c6af-4b88-ba1f-566e2f755881-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-759986c4-v7w8p\" (UID: \"1874e8b8-c6af-4b88-ba1f-566e2f755881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p"
Sep 16 19:12:32.886694 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:32.886581    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1874e8b8-c6af-4b88-ba1f-566e2f755881-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-759986c4-v7w8p\" (UID: \"1874e8b8-c6af-4b88-ba1f-566e2f755881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p"
Sep 16 19:12:32.886694 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:32.886616    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdfsg\" (UniqueName: \"kubernetes.io/projected/1874e8b8-c6af-4b88-ba1f-566e2f755881-kube-api-access-sdfsg\") pod \"isvc-sklearn-runtime-predictor-759986c4-v7w8p\" (UID: \"1874e8b8-c6af-4b88-ba1f-566e2f755881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p"
Sep 16 19:12:32.886965 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:32.886942    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1874e8b8-c6af-4b88-ba1f-566e2f755881-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-759986c4-v7w8p\" (UID: \"1874e8b8-c6af-4b88-ba1f-566e2f755881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p"
Sep 16 19:12:32.887203 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:32.887184    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1874e8b8-c6af-4b88-ba1f-566e2f755881-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-759986c4-v7w8p\" (UID: \"1874e8b8-c6af-4b88-ba1f-566e2f755881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p"
Sep 16 19:12:32.887353 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:32.887333    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1874e8b8-c6af-4b88-ba1f-566e2f755881-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-759986c4-v7w8p\" (UID: \"1874e8b8-c6af-4b88-ba1f-566e2f755881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p"
Sep 16 19:12:32.888977 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:32.888961    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1874e8b8-c6af-4b88-ba1f-566e2f755881-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-759986c4-v7w8p\" (UID: \"1874e8b8-c6af-4b88-ba1f-566e2f755881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p"
Sep 16 19:12:32.896190 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:32.896173    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdfsg\" (UniqueName: \"kubernetes.io/projected/1874e8b8-c6af-4b88-ba1f-566e2f755881-kube-api-access-sdfsg\") pod \"isvc-sklearn-runtime-predictor-759986c4-v7w8p\" (UID: \"1874e8b8-c6af-4b88-ba1f-566e2f755881\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p"
Sep 16 19:12:33.021125 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.021093    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p"
Sep 16 19:12:33.231262 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.231237    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"
Sep 16 19:12:33.289572 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.289512    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-kube-rbac-proxy-sar-config\") pod \"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34\" (UID: \"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34\") "
Sep 16 19:12:33.289686 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.289575    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6md9t\" (UniqueName: \"kubernetes.io/projected/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-kube-api-access-6md9t\") pod \"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34\" (UID: \"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34\") "
Sep 16 19:12:33.289686 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.289620    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-cabundle-cert\") pod \"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34\" (UID: \"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34\") "
Sep 16 19:12:33.289686 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.289638    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-proxy-tls\") pod \"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34\" (UID: \"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34\") "
Sep 16 19:12:33.289686 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.289655    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-kserve-provision-location\") pod \"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34\" (UID: \"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34\") "
Sep 16 19:12:33.289981 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.289955    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "746b0f1d-2fa8-4aaa-bf38-f3f50e631c34" (UID: "746b0f1d-2fa8-4aaa-bf38-f3f50e631c34"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:12:33.290044 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.289947    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "746b0f1d-2fa8-4aaa-bf38-f3f50e631c34" (UID: "746b0f1d-2fa8-4aaa-bf38-f3f50e631c34"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:12:33.290044 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.289984    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "746b0f1d-2fa8-4aaa-bf38-f3f50e631c34" (UID: "746b0f1d-2fa8-4aaa-bf38-f3f50e631c34"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:12:33.291598 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.291577    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "746b0f1d-2fa8-4aaa-bf38-f3f50e631c34" (UID: "746b0f1d-2fa8-4aaa-bf38-f3f50e631c34"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:12:33.291656 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.291643    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-kube-api-access-6md9t" (OuterVolumeSpecName: "kube-api-access-6md9t") pod "746b0f1d-2fa8-4aaa-bf38-f3f50e631c34" (UID: "746b0f1d-2fa8-4aaa-bf38-f3f50e631c34"). InnerVolumeSpecName "kube-api-access-6md9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:12:33.297493 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.297470    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p"]
Sep 16 19:12:33.300014 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:12:33.299995    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1874e8b8_c6af_4b88_ba1f_566e2f755881.slice/crio-738a32f1334afca96655eb88915676d0b87aa4b4d3243bd18bd26853bc2a242e WatchSource:0}: Error finding container 738a32f1334afca96655eb88915676d0b87aa4b4d3243bd18bd26853bc2a242e: Status 404 returned error can't find the container with id 738a32f1334afca96655eb88915676d0b87aa4b4d3243bd18bd26853bc2a242e
Sep 16 19:12:33.390954 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.390928    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6md9t\" (UniqueName: \"kubernetes.io/projected/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-kube-api-access-6md9t\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:12:33.390954 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.390951    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:12:33.391085 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.390964    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:12:33.391085 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.390977    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:12:33.391085 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.390988    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:12:33.493188 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.493159    2554 generic.go:356] "Generic (PLEG): container finished" podID="746b0f1d-2fa8-4aaa-bf38-f3f50e631c34" containerID="3ee9395a8c080e08bd222ba9e65e5ed505aa408a35ec4f44f4c3d63be4801600" exitCode=0
Sep 16 19:12:33.493318 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.493219    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"
Sep 16 19:12:33.493318 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.493225    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676" event={"ID":"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34","Type":"ContainerDied","Data":"3ee9395a8c080e08bd222ba9e65e5ed505aa408a35ec4f44f4c3d63be4801600"}
Sep 16 19:12:33.493318 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.493277    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676" event={"ID":"746b0f1d-2fa8-4aaa-bf38-f3f50e631c34","Type":"ContainerDied","Data":"55fbe50a80645c6560ddaaf0328da2e02ce0cf162e5a9aa1ef80ae47069be16d"}
Sep 16 19:12:33.493318 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.493310    2554 scope.go:117] "RemoveContainer" containerID="a6178eff833b8ea9cdda1a891b1f37c9e104aaa5c2ec9a3e05257c3fcc18f901"
Sep 16 19:12:33.495028 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.495007    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p" event={"ID":"1874e8b8-c6af-4b88-ba1f-566e2f755881","Type":"ContainerStarted","Data":"9e666dd21b891c74fb623781fbb46bc929f20572c065797301a32a225dcb19c6"}
Sep 16 19:12:33.495225 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.495061    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p" event={"ID":"1874e8b8-c6af-4b88-ba1f-566e2f755881","Type":"ContainerStarted","Data":"738a32f1334afca96655eb88915676d0b87aa4b4d3243bd18bd26853bc2a242e"}
Sep 16 19:12:33.503480 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.503459    2554 scope.go:117] "RemoveContainer" containerID="3ee9395a8c080e08bd222ba9e65e5ed505aa408a35ec4f44f4c3d63be4801600"
Sep 16 19:12:33.513973 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.513955    2554 scope.go:117] "RemoveContainer" containerID="d585c9b5d5b1a85c64f8f7143ee5ea4ea00a090772c7f47c29658868012077e0"
Sep 16 19:12:33.522281 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.522262    2554 scope.go:117] "RemoveContainer" containerID="a6178eff833b8ea9cdda1a891b1f37c9e104aaa5c2ec9a3e05257c3fcc18f901"
Sep 16 19:12:33.522558 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:12:33.522542    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6178eff833b8ea9cdda1a891b1f37c9e104aaa5c2ec9a3e05257c3fcc18f901\": container with ID starting with a6178eff833b8ea9cdda1a891b1f37c9e104aaa5c2ec9a3e05257c3fcc18f901 not found: ID does not exist" containerID="a6178eff833b8ea9cdda1a891b1f37c9e104aaa5c2ec9a3e05257c3fcc18f901"
Sep 16 19:12:33.522623 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.522565    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6178eff833b8ea9cdda1a891b1f37c9e104aaa5c2ec9a3e05257c3fcc18f901"} err="failed to get container status \"a6178eff833b8ea9cdda1a891b1f37c9e104aaa5c2ec9a3e05257c3fcc18f901\": rpc error: code = NotFound desc = could not find container \"a6178eff833b8ea9cdda1a891b1f37c9e104aaa5c2ec9a3e05257c3fcc18f901\": container with ID starting with a6178eff833b8ea9cdda1a891b1f37c9e104aaa5c2ec9a3e05257c3fcc18f901 not found: ID does not exist"
Sep 16 19:12:33.522623 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.522582    2554 scope.go:117] "RemoveContainer" containerID="3ee9395a8c080e08bd222ba9e65e5ed505aa408a35ec4f44f4c3d63be4801600"
Sep 16 19:12:33.522875 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:12:33.522849    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ee9395a8c080e08bd222ba9e65e5ed505aa408a35ec4f44f4c3d63be4801600\": container with ID starting with 3ee9395a8c080e08bd222ba9e65e5ed505aa408a35ec4f44f4c3d63be4801600 not found: ID does not exist" containerID="3ee9395a8c080e08bd222ba9e65e5ed505aa408a35ec4f44f4c3d63be4801600"
Sep 16 19:12:33.522958 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.522883    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ee9395a8c080e08bd222ba9e65e5ed505aa408a35ec4f44f4c3d63be4801600"} err="failed to get container status \"3ee9395a8c080e08bd222ba9e65e5ed505aa408a35ec4f44f4c3d63be4801600\": rpc error: code = NotFound desc = could not find container \"3ee9395a8c080e08bd222ba9e65e5ed505aa408a35ec4f44f4c3d63be4801600\": container with ID starting with 3ee9395a8c080e08bd222ba9e65e5ed505aa408a35ec4f44f4c3d63be4801600 not found: ID does not exist"
Sep 16 19:12:33.522958 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.522904    2554 scope.go:117] "RemoveContainer" containerID="d585c9b5d5b1a85c64f8f7143ee5ea4ea00a090772c7f47c29658868012077e0"
Sep 16 19:12:33.523207 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:12:33.523185    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d585c9b5d5b1a85c64f8f7143ee5ea4ea00a090772c7f47c29658868012077e0\": container with ID starting with d585c9b5d5b1a85c64f8f7143ee5ea4ea00a090772c7f47c29658868012077e0 not found: ID does not exist" containerID="d585c9b5d5b1a85c64f8f7143ee5ea4ea00a090772c7f47c29658868012077e0"
Sep 16 19:12:33.523286 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.523217    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d585c9b5d5b1a85c64f8f7143ee5ea4ea00a090772c7f47c29658868012077e0"} err="failed to get container status \"d585c9b5d5b1a85c64f8f7143ee5ea4ea00a090772c7f47c29658868012077e0\": rpc error: code = NotFound desc = could not find container \"d585c9b5d5b1a85c64f8f7143ee5ea4ea00a090772c7f47c29658868012077e0\": container with ID starting with d585c9b5d5b1a85c64f8f7143ee5ea4ea00a090772c7f47c29658868012077e0 not found: ID does not exist"
Sep 16 19:12:33.530380 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.530358    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"]
Sep 16 19:12:33.535585 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:33.535565    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-7z676"]
Sep 16 19:12:34.444165 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:34.444125    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="746b0f1d-2fa8-4aaa-bf38-f3f50e631c34" path="/var/lib/kubelet/pods/746b0f1d-2fa8-4aaa-bf38-f3f50e631c34/volumes"
Sep 16 19:12:39.527784 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:39.527754    2554 generic.go:356] "Generic (PLEG): container finished" podID="1874e8b8-c6af-4b88-ba1f-566e2f755881" containerID="9e666dd21b891c74fb623781fbb46bc929f20572c065797301a32a225dcb19c6" exitCode=0
Sep 16 19:12:39.528098 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:39.527829    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p" event={"ID":"1874e8b8-c6af-4b88-ba1f-566e2f755881","Type":"ContainerDied","Data":"9e666dd21b891c74fb623781fbb46bc929f20572c065797301a32a225dcb19c6"}
Sep 16 19:12:40.535577 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:40.535531    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p" event={"ID":"1874e8b8-c6af-4b88-ba1f-566e2f755881","Type":"ContainerStarted","Data":"8f1fa72e57f38b6dfd3052eae651f4091a72457937a563a3b7396dd9acbcca79"}
Sep 16 19:12:40.535956 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:40.535611    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p" event={"ID":"1874e8b8-c6af-4b88-ba1f-566e2f755881","Type":"ContainerStarted","Data":"00222dd46df193dba302d1aa6e36cb1d14e3d0df87677105a8ac1ca87a3a48a9"}
Sep 16 19:12:40.535956 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:40.535838    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p"
Sep 16 19:12:40.558929 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:40.558874    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p" podStartSLOduration=2.493266586 podStartE2EDuration="8.558855291s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.065588705s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:12:40.554883587 +0000 UTC m=+1670.630653219" watchObservedRunningTime="2026-09-16 19:12:40.558855291 +0000 UTC m=+1670.634624925"
Sep 16 19:12:41.541570 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:41.541538    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p"
Sep 16 19:12:41.543059 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:41.543031    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p" podUID="1874e8b8-c6af-4b88-ba1f-566e2f755881" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:12:42.546092 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:42.546057    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p" podUID="1874e8b8-c6af-4b88-ba1f-566e2f755881" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:12:47.551314 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:47.551284    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p"
Sep 16 19:12:47.558113 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:47.558070    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p" podUID="1874e8b8-c6af-4b88-ba1f-566e2f755881" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Sep 16 19:12:57.552402 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:12:57.552366    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p"
Sep 16 19:13:04.678352 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:04.678321    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p"]
Sep 16 19:13:04.678826 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:04.678640    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p" podUID="1874e8b8-c6af-4b88-ba1f-566e2f755881" containerName="kserve-container" containerID="cri-o://00222dd46df193dba302d1aa6e36cb1d14e3d0df87677105a8ac1ca87a3a48a9" gracePeriod=30
Sep 16 19:13:04.678826 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:04.678672    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p" podUID="1874e8b8-c6af-4b88-ba1f-566e2f755881" containerName="kube-rbac-proxy" containerID="cri-o://8f1fa72e57f38b6dfd3052eae651f4091a72457937a563a3b7396dd9acbcca79" gracePeriod=30
Sep 16 19:13:05.667802 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:05.667768    2554 generic.go:356] "Generic (PLEG): container finished" podID="1874e8b8-c6af-4b88-ba1f-566e2f755881" containerID="8f1fa72e57f38b6dfd3052eae651f4091a72457937a563a3b7396dd9acbcca79" exitCode=2
Sep 16 19:13:05.667802 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:05.667801    2554 generic.go:356] "Generic (PLEG): container finished" podID="1874e8b8-c6af-4b88-ba1f-566e2f755881" containerID="00222dd46df193dba302d1aa6e36cb1d14e3d0df87677105a8ac1ca87a3a48a9" exitCode=0
Sep 16 19:13:05.667987 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:05.667838    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p" event={"ID":"1874e8b8-c6af-4b88-ba1f-566e2f755881","Type":"ContainerDied","Data":"8f1fa72e57f38b6dfd3052eae651f4091a72457937a563a3b7396dd9acbcca79"}
Sep 16 19:13:05.667987 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:05.667902    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p" event={"ID":"1874e8b8-c6af-4b88-ba1f-566e2f755881","Type":"ContainerDied","Data":"00222dd46df193dba302d1aa6e36cb1d14e3d0df87677105a8ac1ca87a3a48a9"}
Sep 16 19:13:05.715163 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:05.715134    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p"
Sep 16 19:13:05.851624 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:05.851564    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1874e8b8-c6af-4b88-ba1f-566e2f755881-kserve-provision-location\") pod \"1874e8b8-c6af-4b88-ba1f-566e2f755881\" (UID: \"1874e8b8-c6af-4b88-ba1f-566e2f755881\") "
Sep 16 19:13:05.851624 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:05.851604    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1874e8b8-c6af-4b88-ba1f-566e2f755881-kube-rbac-proxy-sar-config\") pod \"1874e8b8-c6af-4b88-ba1f-566e2f755881\" (UID: \"1874e8b8-c6af-4b88-ba1f-566e2f755881\") "
Sep 16 19:13:05.851826 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:05.851650    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1874e8b8-c6af-4b88-ba1f-566e2f755881-cabundle-cert\") pod \"1874e8b8-c6af-4b88-ba1f-566e2f755881\" (UID: \"1874e8b8-c6af-4b88-ba1f-566e2f755881\") "
Sep 16 19:13:05.851826 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:05.851684    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1874e8b8-c6af-4b88-ba1f-566e2f755881-proxy-tls\") pod \"1874e8b8-c6af-4b88-ba1f-566e2f755881\" (UID: \"1874e8b8-c6af-4b88-ba1f-566e2f755881\") "
Sep 16 19:13:05.851826 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:05.851754    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sdfsg\" (UniqueName: \"kubernetes.io/projected/1874e8b8-c6af-4b88-ba1f-566e2f755881-kube-api-access-sdfsg\") pod \"1874e8b8-c6af-4b88-ba1f-566e2f755881\" (UID: \"1874e8b8-c6af-4b88-ba1f-566e2f755881\") "
Sep 16 19:13:05.852076 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:05.852053    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1874e8b8-c6af-4b88-ba1f-566e2f755881-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1874e8b8-c6af-4b88-ba1f-566e2f755881" (UID: "1874e8b8-c6af-4b88-ba1f-566e2f755881"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:05.852076 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:05.852066    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1874e8b8-c6af-4b88-ba1f-566e2f755881-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1874e8b8-c6af-4b88-ba1f-566e2f755881" (UID: "1874e8b8-c6af-4b88-ba1f-566e2f755881"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:13:05.853998 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:05.853975    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1874e8b8-c6af-4b88-ba1f-566e2f755881-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1874e8b8-c6af-4b88-ba1f-566e2f755881" (UID: "1874e8b8-c6af-4b88-ba1f-566e2f755881"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:13:05.854171 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:05.854146    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1874e8b8-c6af-4b88-ba1f-566e2f755881-kube-api-access-sdfsg" (OuterVolumeSpecName: "kube-api-access-sdfsg") pod "1874e8b8-c6af-4b88-ba1f-566e2f755881" (UID: "1874e8b8-c6af-4b88-ba1f-566e2f755881"). InnerVolumeSpecName "kube-api-access-sdfsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:13:05.879298 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:05.879276    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1874e8b8-c6af-4b88-ba1f-566e2f755881-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1874e8b8-c6af-4b88-ba1f-566e2f755881" (UID: "1874e8b8-c6af-4b88-ba1f-566e2f755881"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:13:05.952312 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:05.952289    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdfsg\" (UniqueName: \"kubernetes.io/projected/1874e8b8-c6af-4b88-ba1f-566e2f755881-kube-api-access-sdfsg\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:13:05.952312 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:05.952310    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1874e8b8-c6af-4b88-ba1f-566e2f755881-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:13:05.952456 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:05.952319    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1874e8b8-c6af-4b88-ba1f-566e2f755881-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:13:05.952456 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:05.952328    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1874e8b8-c6af-4b88-ba1f-566e2f755881-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:13:05.952456 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:05.952336    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1874e8b8-c6af-4b88-ba1f-566e2f755881-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:13:06.675783 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:06.675739    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p" event={"ID":"1874e8b8-c6af-4b88-ba1f-566e2f755881","Type":"ContainerDied","Data":"738a32f1334afca96655eb88915676d0b87aa4b4d3243bd18bd26853bc2a242e"}
Sep 16 19:13:06.675943 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:06.675799    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p"
Sep 16 19:13:06.675943 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:06.675823    2554 scope.go:117] "RemoveContainer" containerID="8f1fa72e57f38b6dfd3052eae651f4091a72457937a563a3b7396dd9acbcca79"
Sep 16 19:13:06.684542 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:06.684527    2554 scope.go:117] "RemoveContainer" containerID="00222dd46df193dba302d1aa6e36cb1d14e3d0df87677105a8ac1ca87a3a48a9"
Sep 16 19:13:06.692243 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:06.692209    2554 scope.go:117] "RemoveContainer" containerID="9e666dd21b891c74fb623781fbb46bc929f20572c065797301a32a225dcb19c6"
Sep 16 19:13:06.694952 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:06.694934    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p"]
Sep 16 19:13:06.701083 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:06.701029    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-759986c4-v7w8p"]
Sep 16 19:13:08.444022 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:08.443978    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1874e8b8-c6af-4b88-ba1f-566e2f755881" path="/var/lib/kubelet/pods/1874e8b8-c6af-4b88-ba1f-566e2f755881/volumes"
Sep 16 19:13:09.871381 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.871349    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh"]
Sep 16 19:13:09.871775 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.871673    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="746b0f1d-2fa8-4aaa-bf38-f3f50e631c34" containerName="kserve-container"
Sep 16 19:13:09.871775 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.871684    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="746b0f1d-2fa8-4aaa-bf38-f3f50e631c34" containerName="kserve-container"
Sep 16 19:13:09.871775 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.871691    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1874e8b8-c6af-4b88-ba1f-566e2f755881" containerName="kserve-container"
Sep 16 19:13:09.871775 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.871697    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1874e8b8-c6af-4b88-ba1f-566e2f755881" containerName="kserve-container"
Sep 16 19:13:09.871775 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.871706    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="746b0f1d-2fa8-4aaa-bf38-f3f50e631c34" containerName="kube-rbac-proxy"
Sep 16 19:13:09.871775 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.871712    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="746b0f1d-2fa8-4aaa-bf38-f3f50e631c34" containerName="kube-rbac-proxy"
Sep 16 19:13:09.871775 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.871733    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="746b0f1d-2fa8-4aaa-bf38-f3f50e631c34" containerName="storage-initializer"
Sep 16 19:13:09.871775 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.871739    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="746b0f1d-2fa8-4aaa-bf38-f3f50e631c34" containerName="storage-initializer"
Sep 16 19:13:09.871775 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.871745    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1874e8b8-c6af-4b88-ba1f-566e2f755881" containerName="kube-rbac-proxy"
Sep 16 19:13:09.871775 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.871750    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1874e8b8-c6af-4b88-ba1f-566e2f755881" containerName="kube-rbac-proxy"
Sep 16 19:13:09.871775 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.871762    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1874e8b8-c6af-4b88-ba1f-566e2f755881" containerName="storage-initializer"
Sep 16 19:13:09.871775 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.871769    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1874e8b8-c6af-4b88-ba1f-566e2f755881" containerName="storage-initializer"
Sep 16 19:13:09.872157 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.871826    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1874e8b8-c6af-4b88-ba1f-566e2f755881" containerName="kserve-container"
Sep 16 19:13:09.872157 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.871834    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="746b0f1d-2fa8-4aaa-bf38-f3f50e631c34" containerName="kube-rbac-proxy"
Sep 16 19:13:09.872157 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.871839    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1874e8b8-c6af-4b88-ba1f-566e2f755881" containerName="kube-rbac-proxy"
Sep 16 19:13:09.872157 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.871845    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="746b0f1d-2fa8-4aaa-bf38-f3f50e631c34" containerName="kserve-container"
Sep 16 19:13:09.875438 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.875421    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh"
Sep 16 19:13:09.877895 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.877870    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:13:09.878146 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.878130    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Sep 16 19:13:09.878265 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.878249    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:13:09.878444 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.878432    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:13:09.878694 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.878679    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:13:09.878777 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.878685    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8gklb\""
Sep 16 19:13:09.879099 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.879084    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:13:09.890468 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.890449    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh"]
Sep 16 19:13:09.984593 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.984570    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxf5p\" (UniqueName: \"kubernetes.io/projected/e946002d-42b4-430a-85f1-ba4c4108d6b0-kube-api-access-dxf5p\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh\" (UID: \"e946002d-42b4-430a-85f1-ba4c4108d6b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh"
Sep 16 19:13:09.984705 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.984600    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e946002d-42b4-430a-85f1-ba4c4108d6b0-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh\" (UID: \"e946002d-42b4-430a-85f1-ba4c4108d6b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh"
Sep 16 19:13:09.984705 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.984627    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e946002d-42b4-430a-85f1-ba4c4108d6b0-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh\" (UID: \"e946002d-42b4-430a-85f1-ba4c4108d6b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh"
Sep 16 19:13:09.984854 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.984739    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e946002d-42b4-430a-85f1-ba4c4108d6b0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh\" (UID: \"e946002d-42b4-430a-85f1-ba4c4108d6b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh"
Sep 16 19:13:09.984854 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:09.984834    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e946002d-42b4-430a-85f1-ba4c4108d6b0-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh\" (UID: \"e946002d-42b4-430a-85f1-ba4c4108d6b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh"
Sep 16 19:13:10.085936 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:10.085896    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e946002d-42b4-430a-85f1-ba4c4108d6b0-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh\" (UID: \"e946002d-42b4-430a-85f1-ba4c4108d6b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh"
Sep 16 19:13:10.086068 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:10.085960    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxf5p\" (UniqueName: \"kubernetes.io/projected/e946002d-42b4-430a-85f1-ba4c4108d6b0-kube-api-access-dxf5p\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh\" (UID: \"e946002d-42b4-430a-85f1-ba4c4108d6b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh"
Sep 16 19:13:10.086068 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:10.086000    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e946002d-42b4-430a-85f1-ba4c4108d6b0-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh\" (UID: \"e946002d-42b4-430a-85f1-ba4c4108d6b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh"
Sep 16 19:13:10.086068 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:10.086056    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e946002d-42b4-430a-85f1-ba4c4108d6b0-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh\" (UID: \"e946002d-42b4-430a-85f1-ba4c4108d6b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh"
Sep 16 19:13:10.086240 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:10.086096    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e946002d-42b4-430a-85f1-ba4c4108d6b0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh\" (UID: \"e946002d-42b4-430a-85f1-ba4c4108d6b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh"
Sep 16 19:13:10.086510 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:10.086488    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e946002d-42b4-430a-85f1-ba4c4108d6b0-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh\" (UID: \"e946002d-42b4-430a-85f1-ba4c4108d6b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh"
Sep 16 19:13:10.086856 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:10.086804    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e946002d-42b4-430a-85f1-ba4c4108d6b0-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh\" (UID: \"e946002d-42b4-430a-85f1-ba4c4108d6b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh"
Sep 16 19:13:10.086979 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:10.086955    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e946002d-42b4-430a-85f1-ba4c4108d6b0-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh\" (UID: \"e946002d-42b4-430a-85f1-ba4c4108d6b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh"
Sep 16 19:13:10.088579 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:10.088554    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e946002d-42b4-430a-85f1-ba4c4108d6b0-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh\" (UID: \"e946002d-42b4-430a-85f1-ba4c4108d6b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh"
Sep 16 19:13:10.095566 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:10.095545    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxf5p\" (UniqueName: \"kubernetes.io/projected/e946002d-42b4-430a-85f1-ba4c4108d6b0-kube-api-access-dxf5p\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh\" (UID: \"e946002d-42b4-430a-85f1-ba4c4108d6b0\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh"
Sep 16 19:13:10.186130 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:10.186074    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh"
Sep 16 19:13:10.463679 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:10.463659    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh"]
Sep 16 19:13:10.465944 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:13:10.465916    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode946002d_42b4_430a_85f1_ba4c4108d6b0.slice/crio-00bf8cf0941f87137c8c14059c0481908e086b83246c03005327150ff2e6552a WatchSource:0}: Error finding container 00bf8cf0941f87137c8c14059c0481908e086b83246c03005327150ff2e6552a: Status 404 returned error can't find the container with id 00bf8cf0941f87137c8c14059c0481908e086b83246c03005327150ff2e6552a
Sep 16 19:13:10.697519 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:10.697482    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh" event={"ID":"e946002d-42b4-430a-85f1-ba4c4108d6b0","Type":"ContainerStarted","Data":"44e1985b8aef2e2551505df1fb607264acb4d66231a266ee15d451b1ac03e0ad"}
Sep 16 19:13:10.697652 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:10.697557    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh" event={"ID":"e946002d-42b4-430a-85f1-ba4c4108d6b0","Type":"ContainerStarted","Data":"00bf8cf0941f87137c8c14059c0481908e086b83246c03005327150ff2e6552a"}
Sep 16 19:13:14.718098 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:14.718059    2554 generic.go:356] "Generic (PLEG): container finished" podID="e946002d-42b4-430a-85f1-ba4c4108d6b0" containerID="44e1985b8aef2e2551505df1fb607264acb4d66231a266ee15d451b1ac03e0ad" exitCode=0
Sep 16 19:13:14.718490 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:14.718099    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh" event={"ID":"e946002d-42b4-430a-85f1-ba4c4108d6b0","Type":"ContainerDied","Data":"44e1985b8aef2e2551505df1fb607264acb4d66231a266ee15d451b1ac03e0ad"}
Sep 16 19:13:15.724461 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:15.724426    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh" event={"ID":"e946002d-42b4-430a-85f1-ba4c4108d6b0","Type":"ContainerStarted","Data":"7f44eca06809c453888ba1d5c6bcab14aab207fad99f52ef2c1ba4c4e96e9cd1"}
Sep 16 19:13:15.724840 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:15.724508    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh" event={"ID":"e946002d-42b4-430a-85f1-ba4c4108d6b0","Type":"ContainerStarted","Data":"56ada083f588506ec444b573d790ccf7be3e6a69f5861448a106f0d2d6d813fe"}
Sep 16 19:13:15.724840 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:15.724695    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh"
Sep 16 19:13:15.744049 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:15.743997    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh" podStartSLOduration=2.672233601 podStartE2EDuration="6.743981096s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.071747495s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:13:15.743420487 +0000 UTC m=+1705.819190119" watchObservedRunningTime="2026-09-16 19:13:15.743981096 +0000 UTC m=+1705.819750728"
Sep 16 19:13:16.728368 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:16.728340    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh"
Sep 16 19:13:22.738657 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:22.738629    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh"
Sep 16 19:13:52.751525 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:13:52.751481    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh" podUID="e946002d-42b4-430a-85f1-ba4c4108d6b0" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 16 19:14:02.741513 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:02.741483    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh"
Sep 16 19:14:09.986629 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:09.983734    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh"]
Sep 16 19:14:09.986629 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:09.984086    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh" podUID="e946002d-42b4-430a-85f1-ba4c4108d6b0" containerName="kserve-container" containerID="cri-o://56ada083f588506ec444b573d790ccf7be3e6a69f5861448a106f0d2d6d813fe" gracePeriod=30
Sep 16 19:14:09.986629 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:09.984314    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh" podUID="e946002d-42b4-430a-85f1-ba4c4108d6b0" containerName="kube-rbac-proxy" containerID="cri-o://7f44eca06809c453888ba1d5c6bcab14aab207fad99f52ef2c1ba4c4e96e9cd1" gracePeriod=30
Sep 16 19:14:10.978245 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:10.978213    2554 generic.go:356] "Generic (PLEG): container finished" podID="e946002d-42b4-430a-85f1-ba4c4108d6b0" containerID="7f44eca06809c453888ba1d5c6bcab14aab207fad99f52ef2c1ba4c4e96e9cd1" exitCode=2
Sep 16 19:14:10.978415 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:10.978278    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh" event={"ID":"e946002d-42b4-430a-85f1-ba4c4108d6b0","Type":"ContainerDied","Data":"7f44eca06809c453888ba1d5c6bcab14aab207fad99f52ef2c1ba4c4e96e9cd1"}
Sep 16 19:14:12.733903 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:12.733862    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh" podUID="e946002d-42b4-430a-85f1-ba4c4108d6b0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Sep 16 19:14:12.739769 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:12.739739    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh" podUID="e946002d-42b4-430a-85f1-ba4c4108d6b0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.43:8080/v2/models/isvc-sklearn-v2-runtime/ready\": dial tcp 10.134.0.43:8080: connect: connection refused"
Sep 16 19:14:15.112500 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.112463    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"]
Sep 16 19:14:15.114845 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.114829    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"
Sep 16 19:14:15.118813 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.118785    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Sep 16 19:14:15.118932 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.118885    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 16 19:14:15.141438 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.141408    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"]
Sep 16 19:14:15.167113 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.167084    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4710141-8492-43c6-a2b3-b7b48a7272d6-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-ccf59f694-5ftbg\" (UID: \"b4710141-8492-43c6-a2b3-b7b48a7272d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"
Sep 16 19:14:15.167246 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.167139    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4710141-8492-43c6-a2b3-b7b48a7272d6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-ccf59f694-5ftbg\" (UID: \"b4710141-8492-43c6-a2b3-b7b48a7272d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"
Sep 16 19:14:15.167246 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.167179    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4710141-8492-43c6-a2b3-b7b48a7272d6-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-ccf59f694-5ftbg\" (UID: \"b4710141-8492-43c6-a2b3-b7b48a7272d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"
Sep 16 19:14:15.167336 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.167276    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4710141-8492-43c6-a2b3-b7b48a7272d6-proxy-tls\") pod \"isvc-sklearn-v2-predictor-ccf59f694-5ftbg\" (UID: \"b4710141-8492-43c6-a2b3-b7b48a7272d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"
Sep 16 19:14:15.167382 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.167337    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvfxg\" (UniqueName: \"kubernetes.io/projected/b4710141-8492-43c6-a2b3-b7b48a7272d6-kube-api-access-kvfxg\") pod \"isvc-sklearn-v2-predictor-ccf59f694-5ftbg\" (UID: \"b4710141-8492-43c6-a2b3-b7b48a7272d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"
Sep 16 19:14:15.268298 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.268275    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4710141-8492-43c6-a2b3-b7b48a7272d6-proxy-tls\") pod \"isvc-sklearn-v2-predictor-ccf59f694-5ftbg\" (UID: \"b4710141-8492-43c6-a2b3-b7b48a7272d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"
Sep 16 19:14:15.268401 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.268316    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvfxg\" (UniqueName: \"kubernetes.io/projected/b4710141-8492-43c6-a2b3-b7b48a7272d6-kube-api-access-kvfxg\") pod \"isvc-sklearn-v2-predictor-ccf59f694-5ftbg\" (UID: \"b4710141-8492-43c6-a2b3-b7b48a7272d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"
Sep 16 19:14:15.268401 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.268359    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4710141-8492-43c6-a2b3-b7b48a7272d6-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-ccf59f694-5ftbg\" (UID: \"b4710141-8492-43c6-a2b3-b7b48a7272d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"
Sep 16 19:14:15.268401 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.268390    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4710141-8492-43c6-a2b3-b7b48a7272d6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-ccf59f694-5ftbg\" (UID: \"b4710141-8492-43c6-a2b3-b7b48a7272d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"
Sep 16 19:14:15.268527 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.268404    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4710141-8492-43c6-a2b3-b7b48a7272d6-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-ccf59f694-5ftbg\" (UID: \"b4710141-8492-43c6-a2b3-b7b48a7272d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"
Sep 16 19:14:15.268527 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:14:15.268426    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-predictor-serving-cert: secret "isvc-sklearn-v2-predictor-serving-cert" not found
Sep 16 19:14:15.268527 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:14:15.268508    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4710141-8492-43c6-a2b3-b7b48a7272d6-proxy-tls podName:b4710141-8492-43c6-a2b3-b7b48a7272d6 nodeName:}" failed. No retries permitted until 2026-09-16 19:14:15.768486811 +0000 UTC m=+1765.844256437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b4710141-8492-43c6-a2b3-b7b48a7272d6-proxy-tls") pod "isvc-sklearn-v2-predictor-ccf59f694-5ftbg" (UID: "b4710141-8492-43c6-a2b3-b7b48a7272d6") : secret "isvc-sklearn-v2-predictor-serving-cert" not found
Sep 16 19:14:15.268776 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.268755    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4710141-8492-43c6-a2b3-b7b48a7272d6-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-ccf59f694-5ftbg\" (UID: \"b4710141-8492-43c6-a2b3-b7b48a7272d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"
Sep 16 19:14:15.269019 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.269002    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4710141-8492-43c6-a2b3-b7b48a7272d6-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-ccf59f694-5ftbg\" (UID: \"b4710141-8492-43c6-a2b3-b7b48a7272d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"
Sep 16 19:14:15.269074 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.269018    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4710141-8492-43c6-a2b3-b7b48a7272d6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-ccf59f694-5ftbg\" (UID: \"b4710141-8492-43c6-a2b3-b7b48a7272d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"
Sep 16 19:14:15.276598 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.276581    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvfxg\" (UniqueName: \"kubernetes.io/projected/b4710141-8492-43c6-a2b3-b7b48a7272d6-kube-api-access-kvfxg\") pod \"isvc-sklearn-v2-predictor-ccf59f694-5ftbg\" (UID: \"b4710141-8492-43c6-a2b3-b7b48a7272d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"
Sep 16 19:14:15.429053 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.429029    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh"
Sep 16 19:14:15.570687 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.570656    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e946002d-42b4-430a-85f1-ba4c4108d6b0-kserve-provision-location\") pod \"e946002d-42b4-430a-85f1-ba4c4108d6b0\" (UID: \"e946002d-42b4-430a-85f1-ba4c4108d6b0\") "
Sep 16 19:14:15.570842 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.570710    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxf5p\" (UniqueName: \"kubernetes.io/projected/e946002d-42b4-430a-85f1-ba4c4108d6b0-kube-api-access-dxf5p\") pod \"e946002d-42b4-430a-85f1-ba4c4108d6b0\" (UID: \"e946002d-42b4-430a-85f1-ba4c4108d6b0\") "
Sep 16 19:14:15.570842 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.570769    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e946002d-42b4-430a-85f1-ba4c4108d6b0-proxy-tls\") pod \"e946002d-42b4-430a-85f1-ba4c4108d6b0\" (UID: \"e946002d-42b4-430a-85f1-ba4c4108d6b0\") "
Sep 16 19:14:15.570842 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.570817    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e946002d-42b4-430a-85f1-ba4c4108d6b0-kube-rbac-proxy-sar-config\") pod \"e946002d-42b4-430a-85f1-ba4c4108d6b0\" (UID: \"e946002d-42b4-430a-85f1-ba4c4108d6b0\") "
Sep 16 19:14:15.570969 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.570853    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e946002d-42b4-430a-85f1-ba4c4108d6b0-cabundle-cert\") pod \"e946002d-42b4-430a-85f1-ba4c4108d6b0\" (UID: \"e946002d-42b4-430a-85f1-ba4c4108d6b0\") "
Sep 16 19:14:15.570969 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.570947    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e946002d-42b4-430a-85f1-ba4c4108d6b0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e946002d-42b4-430a-85f1-ba4c4108d6b0" (UID: "e946002d-42b4-430a-85f1-ba4c4108d6b0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:14:15.571276 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.571239    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e946002d-42b4-430a-85f1-ba4c4108d6b0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e946002d-42b4-430a-85f1-ba4c4108d6b0" (UID: "e946002d-42b4-430a-85f1-ba4c4108d6b0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:15.573200 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.573177    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e946002d-42b4-430a-85f1-ba4c4108d6b0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:14:15.573440 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.573412    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e946002d-42b4-430a-85f1-ba4c4108d6b0-kube-api-access-dxf5p" (OuterVolumeSpecName: "kube-api-access-dxf5p") pod "e946002d-42b4-430a-85f1-ba4c4108d6b0" (UID: "e946002d-42b4-430a-85f1-ba4c4108d6b0"). InnerVolumeSpecName "kube-api-access-dxf5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:14:15.574076 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.571274    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e946002d-42b4-430a-85f1-ba4c4108d6b0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e946002d-42b4-430a-85f1-ba4c4108d6b0" (UID: "e946002d-42b4-430a-85f1-ba4c4108d6b0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:14:15.574195 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.574183    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e946002d-42b4-430a-85f1-ba4c4108d6b0-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:14:15.578761 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.577202    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e946002d-42b4-430a-85f1-ba4c4108d6b0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e946002d-42b4-430a-85f1-ba4c4108d6b0" (UID: "e946002d-42b4-430a-85f1-ba4c4108d6b0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:14:15.675082 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.675021    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxf5p\" (UniqueName: \"kubernetes.io/projected/e946002d-42b4-430a-85f1-ba4c4108d6b0-kube-api-access-dxf5p\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:14:15.675082 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.675051    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e946002d-42b4-430a-85f1-ba4c4108d6b0-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:14:15.675082 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.675060    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e946002d-42b4-430a-85f1-ba4c4108d6b0-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:14:15.776029 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.776004    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4710141-8492-43c6-a2b3-b7b48a7272d6-proxy-tls\") pod \"isvc-sklearn-v2-predictor-ccf59f694-5ftbg\" (UID: \"b4710141-8492-43c6-a2b3-b7b48a7272d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"
Sep 16 19:14:15.778107 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:15.778091    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4710141-8492-43c6-a2b3-b7b48a7272d6-proxy-tls\") pod \"isvc-sklearn-v2-predictor-ccf59f694-5ftbg\" (UID: \"b4710141-8492-43c6-a2b3-b7b48a7272d6\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"
Sep 16 19:14:16.001818 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:16.001797    2554 generic.go:356] "Generic (PLEG): container finished" podID="e946002d-42b4-430a-85f1-ba4c4108d6b0" containerID="56ada083f588506ec444b573d790ccf7be3e6a69f5861448a106f0d2d6d813fe" exitCode=0
Sep 16 19:14:16.001912 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:16.001857    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh"
Sep 16 19:14:16.001912 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:16.001882    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh" event={"ID":"e946002d-42b4-430a-85f1-ba4c4108d6b0","Type":"ContainerDied","Data":"56ada083f588506ec444b573d790ccf7be3e6a69f5861448a106f0d2d6d813fe"}
Sep 16 19:14:16.001982 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:16.001962    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh" event={"ID":"e946002d-42b4-430a-85f1-ba4c4108d6b0","Type":"ContainerDied","Data":"00bf8cf0941f87137c8c14059c0481908e086b83246c03005327150ff2e6552a"}
Sep 16 19:14:16.002019 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:16.001988    2554 scope.go:117] "RemoveContainer" containerID="7f44eca06809c453888ba1d5c6bcab14aab207fad99f52ef2c1ba4c4e96e9cd1"
Sep 16 19:14:16.009934 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:16.009920    2554 scope.go:117] "RemoveContainer" containerID="56ada083f588506ec444b573d790ccf7be3e6a69f5861448a106f0d2d6d813fe"
Sep 16 19:14:16.017065 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:16.017051    2554 scope.go:117] "RemoveContainer" containerID="44e1985b8aef2e2551505df1fb607264acb4d66231a266ee15d451b1ac03e0ad"
Sep 16 19:14:16.023413 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:16.023394    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh"]
Sep 16 19:14:16.024379 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:16.024362    2554 scope.go:117] "RemoveContainer" containerID="7f44eca06809c453888ba1d5c6bcab14aab207fad99f52ef2c1ba4c4e96e9cd1"
Sep 16 19:14:16.024476 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:16.024458    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"
Sep 16 19:14:16.024630 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:14:16.024611    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f44eca06809c453888ba1d5c6bcab14aab207fad99f52ef2c1ba4c4e96e9cd1\": container with ID starting with 7f44eca06809c453888ba1d5c6bcab14aab207fad99f52ef2c1ba4c4e96e9cd1 not found: ID does not exist" containerID="7f44eca06809c453888ba1d5c6bcab14aab207fad99f52ef2c1ba4c4e96e9cd1"
Sep 16 19:14:16.024681 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:16.024636    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f44eca06809c453888ba1d5c6bcab14aab207fad99f52ef2c1ba4c4e96e9cd1"} err="failed to get container status \"7f44eca06809c453888ba1d5c6bcab14aab207fad99f52ef2c1ba4c4e96e9cd1\": rpc error: code = NotFound desc = could not find container \"7f44eca06809c453888ba1d5c6bcab14aab207fad99f52ef2c1ba4c4e96e9cd1\": container with ID starting with 7f44eca06809c453888ba1d5c6bcab14aab207fad99f52ef2c1ba4c4e96e9cd1 not found: ID does not exist"
Sep 16 19:14:16.024681 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:16.024651    2554 scope.go:117] "RemoveContainer" containerID="56ada083f588506ec444b573d790ccf7be3e6a69f5861448a106f0d2d6d813fe"
Sep 16 19:14:16.024886 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:14:16.024871    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ada083f588506ec444b573d790ccf7be3e6a69f5861448a106f0d2d6d813fe\": container with ID starting with 56ada083f588506ec444b573d790ccf7be3e6a69f5861448a106f0d2d6d813fe not found: ID does not exist" containerID="56ada083f588506ec444b573d790ccf7be3e6a69f5861448a106f0d2d6d813fe"
Sep 16 19:14:16.024926 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:16.024891    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ada083f588506ec444b573d790ccf7be3e6a69f5861448a106f0d2d6d813fe"} err="failed to get container status \"56ada083f588506ec444b573d790ccf7be3e6a69f5861448a106f0d2d6d813fe\": rpc error: code = NotFound desc = could not find container \"56ada083f588506ec444b573d790ccf7be3e6a69f5861448a106f0d2d6d813fe\": container with ID starting with 56ada083f588506ec444b573d790ccf7be3e6a69f5861448a106f0d2d6d813fe not found: ID does not exist"
Sep 16 19:14:16.024926 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:16.024903    2554 scope.go:117] "RemoveContainer" containerID="44e1985b8aef2e2551505df1fb607264acb4d66231a266ee15d451b1ac03e0ad"
Sep 16 19:14:16.025094 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:14:16.025081    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44e1985b8aef2e2551505df1fb607264acb4d66231a266ee15d451b1ac03e0ad\": container with ID starting with 44e1985b8aef2e2551505df1fb607264acb4d66231a266ee15d451b1ac03e0ad not found: ID does not exist" containerID="44e1985b8aef2e2551505df1fb607264acb4d66231a266ee15d451b1ac03e0ad"
Sep 16 19:14:16.025171 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:16.025098    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44e1985b8aef2e2551505df1fb607264acb4d66231a266ee15d451b1ac03e0ad"} err="failed to get container status \"44e1985b8aef2e2551505df1fb607264acb4d66231a266ee15d451b1ac03e0ad\": rpc error: code = NotFound desc = could not find container \"44e1985b8aef2e2551505df1fb607264acb4d66231a266ee15d451b1ac03e0ad\": container with ID starting with 44e1985b8aef2e2551505df1fb607264acb4d66231a266ee15d451b1ac03e0ad not found: ID does not exist"
Sep 16 19:14:16.028154 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:16.028137    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-8hvhh"]
Sep 16 19:14:16.293360 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:16.293341    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"]
Sep 16 19:14:16.295524 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:14:16.295498    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4710141_8492_43c6_a2b3_b7b48a7272d6.slice/crio-cec5ef2548e10ed6a85f059151ac2451f895742f77374ccdb7268d136390d5e6 WatchSource:0}: Error finding container cec5ef2548e10ed6a85f059151ac2451f895742f77374ccdb7268d136390d5e6: Status 404 returned error can't find the container with id cec5ef2548e10ed6a85f059151ac2451f895742f77374ccdb7268d136390d5e6
Sep 16 19:14:16.444180 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:16.444139    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e946002d-42b4-430a-85f1-ba4c4108d6b0" path="/var/lib/kubelet/pods/e946002d-42b4-430a-85f1-ba4c4108d6b0/volumes"
Sep 16 19:14:17.008777 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:17.008740    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg" event={"ID":"b4710141-8492-43c6-a2b3-b7b48a7272d6","Type":"ContainerStarted","Data":"41de7ee37a4f765b20bfcad84ca6a396bb9577eb96e9fa98fe5b7462d1cfcd60"}
Sep 16 19:14:17.008937 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:17.008808    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg" event={"ID":"b4710141-8492-43c6-a2b3-b7b48a7272d6","Type":"ContainerStarted","Data":"cec5ef2548e10ed6a85f059151ac2451f895742f77374ccdb7268d136390d5e6"}
Sep 16 19:14:21.026624 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:21.026592    2554 generic.go:356] "Generic (PLEG): container finished" podID="b4710141-8492-43c6-a2b3-b7b48a7272d6" containerID="41de7ee37a4f765b20bfcad84ca6a396bb9577eb96e9fa98fe5b7462d1cfcd60" exitCode=0
Sep 16 19:14:21.026987 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:21.026639    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg" event={"ID":"b4710141-8492-43c6-a2b3-b7b48a7272d6","Type":"ContainerDied","Data":"41de7ee37a4f765b20bfcad84ca6a396bb9577eb96e9fa98fe5b7462d1cfcd60"}
Sep 16 19:14:22.033392 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:22.033360    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg" event={"ID":"b4710141-8492-43c6-a2b3-b7b48a7272d6","Type":"ContainerStarted","Data":"baafe2b873166c4c3e34e355e83f4fbce3cf4104fa478e8c028759ee1dc5ab99"}
Sep 16 19:14:22.033763 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:22.033416    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg" event={"ID":"b4710141-8492-43c6-a2b3-b7b48a7272d6","Type":"ContainerStarted","Data":"92a1ebe9a88facfb936cd54914a6935f01c492075a56c5a0fbafd537acfbddf0"}
Sep 16 19:14:22.033763 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:22.033668    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"
Sep 16 19:14:22.033763 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:22.033710    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"
Sep 16 19:14:22.034694 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:22.034672    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg" podUID="b4710141-8492-43c6-a2b3-b7b48a7272d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:14:22.053769 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:22.053711    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg" podStartSLOduration=3.198182311 podStartE2EDuration="7.053698404s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.855516093s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:22.051454464 +0000 UTC m=+1772.127224096" watchObservedRunningTime="2026-09-16 19:14:22.053698404 +0000 UTC m=+1772.129468035"
Sep 16 19:14:23.038519 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:23.038482    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg" podUID="b4710141-8492-43c6-a2b3-b7b48a7272d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:14:28.042936 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:28.042910    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"
Sep 16 19:14:28.043470 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:28.043438    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg" podUID="b4710141-8492-43c6-a2b3-b7b48a7272d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:14:38.043788 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:38.043753    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg" podUID="b4710141-8492-43c6-a2b3-b7b48a7272d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:14:39.577546 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.577516    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch"]
Sep 16 19:14:39.577904 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.577868    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e946002d-42b4-430a-85f1-ba4c4108d6b0" containerName="kube-rbac-proxy"
Sep 16 19:14:39.577904 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.577880    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e946002d-42b4-430a-85f1-ba4c4108d6b0" containerName="kube-rbac-proxy"
Sep 16 19:14:39.577904 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.577892    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e946002d-42b4-430a-85f1-ba4c4108d6b0" containerName="storage-initializer"
Sep 16 19:14:39.577904 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.577897    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e946002d-42b4-430a-85f1-ba4c4108d6b0" containerName="storage-initializer"
Sep 16 19:14:39.578039 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.577909    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e946002d-42b4-430a-85f1-ba4c4108d6b0" containerName="kserve-container"
Sep 16 19:14:39.578039 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.577915    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e946002d-42b4-430a-85f1-ba4c4108d6b0" containerName="kserve-container"
Sep 16 19:14:39.578039 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.577964    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e946002d-42b4-430a-85f1-ba4c4108d6b0" containerName="kube-rbac-proxy"
Sep 16 19:14:39.578039 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.577977    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e946002d-42b4-430a-85f1-ba4c4108d6b0" containerName="kserve-container"
Sep 16 19:14:39.581083 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.581066    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch"
Sep 16 19:14:39.583178 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.583136    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Sep 16 19:14:39.583292 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.583254    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 16 19:14:39.594751 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.594730    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch"]
Sep 16 19:14:39.624537 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.624515    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct"]
Sep 16 19:14:39.627921 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.627904    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct"
Sep 16 19:14:39.630707 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.630689    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 16 19:14:39.643235 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.643214    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct"]
Sep 16 19:14:39.757321 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.757292    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-proxy-tls\") pod \"isvc-canary-create-predictor-6fb974599-bpsct\" (UID: \"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct"
Sep 16 19:14:39.757435 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.757325    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89vcx\" (UniqueName: \"kubernetes.io/projected/a5b01dce-7cd8-483a-8480-8027abc6fae0-kube-api-access-89vcx\") pod \"isvc-canary-create-v2-predictor-847844f798-qldch\" (UID: \"a5b01dce-7cd8-483a-8480-8027abc6fae0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch"
Sep 16 19:14:39.757435 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.757366    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5b01dce-7cd8-483a-8480-8027abc6fae0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-847844f798-qldch\" (UID: \"a5b01dce-7cd8-483a-8480-8027abc6fae0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch"
Sep 16 19:14:39.757435 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.757395    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-cabundle-cert\") pod \"isvc-canary-create-predictor-6fb974599-bpsct\" (UID: \"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct"
Sep 16 19:14:39.757435 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.757417    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5b01dce-7cd8-483a-8480-8027abc6fae0-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-847844f798-qldch\" (UID: \"a5b01dce-7cd8-483a-8480-8027abc6fae0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch"
Sep 16 19:14:39.757570 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.757469    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6fb974599-bpsct\" (UID: \"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct"
Sep 16 19:14:39.757570 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.757493    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgsbt\" (UniqueName: \"kubernetes.io/projected/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-kube-api-access-cgsbt\") pod \"isvc-canary-create-predictor-6fb974599-bpsct\" (UID: \"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct"
Sep 16 19:14:39.757570 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.757532    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-kserve-provision-location\") pod \"isvc-canary-create-predictor-6fb974599-bpsct\" (UID: \"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct"
Sep 16 19:14:39.757570 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.757549    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5b01dce-7cd8-483a-8480-8027abc6fae0-proxy-tls\") pod \"isvc-canary-create-v2-predictor-847844f798-qldch\" (UID: \"a5b01dce-7cd8-483a-8480-8027abc6fae0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch"
Sep 16 19:14:39.757692 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.757574    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5b01dce-7cd8-483a-8480-8027abc6fae0-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-847844f798-qldch\" (UID: \"a5b01dce-7cd8-483a-8480-8027abc6fae0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch"
Sep 16 19:14:39.858677 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.858618    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5b01dce-7cd8-483a-8480-8027abc6fae0-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-847844f798-qldch\" (UID: \"a5b01dce-7cd8-483a-8480-8027abc6fae0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch"
Sep 16 19:14:39.858677 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.858648    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6fb974599-bpsct\" (UID: \"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct"
Sep 16 19:14:39.858677 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.858676    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgsbt\" (UniqueName: \"kubernetes.io/projected/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-kube-api-access-cgsbt\") pod \"isvc-canary-create-predictor-6fb974599-bpsct\" (UID: \"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct"
Sep 16 19:14:39.858868 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.858699    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-kserve-provision-location\") pod \"isvc-canary-create-predictor-6fb974599-bpsct\" (UID: \"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct"
Sep 16 19:14:39.858868 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.858714    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5b01dce-7cd8-483a-8480-8027abc6fae0-proxy-tls\") pod \"isvc-canary-create-v2-predictor-847844f798-qldch\" (UID: \"a5b01dce-7cd8-483a-8480-8027abc6fae0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch"
Sep 16 19:14:39.858868 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.858755    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5b01dce-7cd8-483a-8480-8027abc6fae0-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-847844f798-qldch\" (UID: \"a5b01dce-7cd8-483a-8480-8027abc6fae0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch"
Sep 16 19:14:39.858868 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.858799    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-proxy-tls\") pod \"isvc-canary-create-predictor-6fb974599-bpsct\" (UID: \"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct"
Sep 16 19:14:39.858868 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.858827    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89vcx\" (UniqueName: \"kubernetes.io/projected/a5b01dce-7cd8-483a-8480-8027abc6fae0-kube-api-access-89vcx\") pod \"isvc-canary-create-v2-predictor-847844f798-qldch\" (UID: \"a5b01dce-7cd8-483a-8480-8027abc6fae0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch"
Sep 16 19:14:39.859132 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.858873    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5b01dce-7cd8-483a-8480-8027abc6fae0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-847844f798-qldch\" (UID: \"a5b01dce-7cd8-483a-8480-8027abc6fae0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch"
Sep 16 19:14:39.859132 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.858911    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-cabundle-cert\") pod \"isvc-canary-create-predictor-6fb974599-bpsct\" (UID: \"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct"
Sep 16 19:14:39.859421 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.859395    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5b01dce-7cd8-483a-8480-8027abc6fae0-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-847844f798-qldch\" (UID: \"a5b01dce-7cd8-483a-8480-8027abc6fae0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch"
Sep 16 19:14:39.859573 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.859428    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5b01dce-7cd8-483a-8480-8027abc6fae0-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-847844f798-qldch\" (UID: \"a5b01dce-7cd8-483a-8480-8027abc6fae0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch"
Sep 16 19:14:39.859698 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.859486    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-kserve-provision-location\") pod \"isvc-canary-create-predictor-6fb974599-bpsct\" (UID: \"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct"
Sep 16 19:14:39.860014 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.859990    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5b01dce-7cd8-483a-8480-8027abc6fae0-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-847844f798-qldch\" (UID: \"a5b01dce-7cd8-483a-8480-8027abc6fae0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch"
Sep 16 19:14:39.860014 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.860001    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-6fb974599-bpsct\" (UID: \"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct"
Sep 16 19:14:39.860134 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.860001    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-cabundle-cert\") pod \"isvc-canary-create-predictor-6fb974599-bpsct\" (UID: \"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct"
Sep 16 19:14:39.861296 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.861277    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5b01dce-7cd8-483a-8480-8027abc6fae0-proxy-tls\") pod \"isvc-canary-create-v2-predictor-847844f798-qldch\" (UID: \"a5b01dce-7cd8-483a-8480-8027abc6fae0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch"
Sep 16 19:14:39.861409 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.861392    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-proxy-tls\") pod \"isvc-canary-create-predictor-6fb974599-bpsct\" (UID: \"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct"
Sep 16 19:14:39.867469 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.867446    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgsbt\" (UniqueName: \"kubernetes.io/projected/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-kube-api-access-cgsbt\") pod \"isvc-canary-create-predictor-6fb974599-bpsct\" (UID: \"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct"
Sep 16 19:14:39.868710 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.868692    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89vcx\" (UniqueName: \"kubernetes.io/projected/a5b01dce-7cd8-483a-8480-8027abc6fae0-kube-api-access-89vcx\") pod \"isvc-canary-create-v2-predictor-847844f798-qldch\" (UID: \"a5b01dce-7cd8-483a-8480-8027abc6fae0\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch"
Sep 16 19:14:39.891471 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.891452    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch"
Sep 16 19:14:39.936928 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:39.936661    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct"
Sep 16 19:14:40.164192 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:40.164117    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch"]
Sep 16 19:14:40.167021 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:14:40.167000    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5b01dce_7cd8_483a_8480_8027abc6fae0.slice/crio-2d8df22d09ffb64d2874a4eae9097e7e40d150305ee1eb4e0fd433dd1e0c2fb5 WatchSource:0}: Error finding container 2d8df22d09ffb64d2874a4eae9097e7e40d150305ee1eb4e0fd433dd1e0c2fb5: Status 404 returned error can't find the container with id 2d8df22d09ffb64d2874a4eae9097e7e40d150305ee1eb4e0fd433dd1e0c2fb5
Sep 16 19:14:40.226171 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:40.226148    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct"]
Sep 16 19:14:40.227555 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:14:40.227529    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b3246b_d1cb_429a_9cc7_f29ea2ed127a.slice/crio-1def290ef9f4ee2972b9b4a5981267450774e439f97bf61a4218a07689521476 WatchSource:0}: Error finding container 1def290ef9f4ee2972b9b4a5981267450774e439f97bf61a4218a07689521476: Status 404 returned error can't find the container with id 1def290ef9f4ee2972b9b4a5981267450774e439f97bf61a4218a07689521476
Sep 16 19:14:41.118777 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:41.118738    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct" event={"ID":"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a","Type":"ContainerStarted","Data":"0489d0dd28f1f1b3fbe99cbfb98e767dbe59186437e722d944dba8235453e70b"}
Sep 16 19:14:41.119230 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:41.118813    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct" event={"ID":"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a","Type":"ContainerStarted","Data":"1def290ef9f4ee2972b9b4a5981267450774e439f97bf61a4218a07689521476"}
Sep 16 19:14:41.120278 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:41.120252    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch" event={"ID":"a5b01dce-7cd8-483a-8480-8027abc6fae0","Type":"ContainerStarted","Data":"2d2f0eed534cb46d12a306459a29641c9113a9c0ce198c646f3f6ca1b2e5ed57"}
Sep 16 19:14:41.120379 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:41.120309    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch" event={"ID":"a5b01dce-7cd8-483a-8480-8027abc6fae0","Type":"ContainerStarted","Data":"2d8df22d09ffb64d2874a4eae9097e7e40d150305ee1eb4e0fd433dd1e0c2fb5"}
Sep 16 19:14:44.134710 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:44.134684    2554 generic.go:356] "Generic (PLEG): container finished" podID="a5b01dce-7cd8-483a-8480-8027abc6fae0" containerID="2d2f0eed534cb46d12a306459a29641c9113a9c0ce198c646f3f6ca1b2e5ed57" exitCode=0
Sep 16 19:14:44.134998 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:44.134884    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch" event={"ID":"a5b01dce-7cd8-483a-8480-8027abc6fae0","Type":"ContainerDied","Data":"2d2f0eed534cb46d12a306459a29641c9113a9c0ce198c646f3f6ca1b2e5ed57"}
Sep 16 19:14:44.136829 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:44.136775    2554 generic.go:356] "Generic (PLEG): container finished" podID="b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" containerID="0489d0dd28f1f1b3fbe99cbfb98e767dbe59186437e722d944dba8235453e70b" exitCode=0
Sep 16 19:14:44.136829 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:44.136813    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct" event={"ID":"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a","Type":"ContainerDied","Data":"0489d0dd28f1f1b3fbe99cbfb98e767dbe59186437e722d944dba8235453e70b"}
Sep 16 19:14:45.144397 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:45.144347    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct" event={"ID":"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a","Type":"ContainerStarted","Data":"de664d5f230ccdf4b3c876ec9dd619db6f79da81c801690fc2a1354bd121b70e"}
Sep 16 19:14:45.144829 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:45.144428    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct" event={"ID":"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a","Type":"ContainerStarted","Data":"e30958675e511eea6934a04123e1157c817e814c56601801c410c24c076b4e62"}
Sep 16 19:14:45.144829 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:45.144626    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct"
Sep 16 19:14:45.146391 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:45.146371    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch" event={"ID":"a5b01dce-7cd8-483a-8480-8027abc6fae0","Type":"ContainerStarted","Data":"477e5855aeb975a0c92bd670bb859956bef5eb8c29582988fd280fe0b732ede3"}
Sep 16 19:14:45.146500 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:45.146412    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch" event={"ID":"a5b01dce-7cd8-483a-8480-8027abc6fae0","Type":"ContainerStarted","Data":"6195e4a54c6fe328d536b4881e611e020feaf1480bbbf7a88b97a477c16b2997"}
Sep 16 19:14:45.146588 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:45.146575    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch"
Sep 16 19:14:45.216775 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:45.216689    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch" podStartSLOduration=2.330423965 podStartE2EDuration="6.216673171s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.886249206s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:45.214420966 +0000 UTC m=+1795.290190596" watchObservedRunningTime="2026-09-16 19:14:45.216673171 +0000 UTC m=+1795.292442792"
Sep 16 19:14:45.216916 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:45.216838    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct" podStartSLOduration=2.616613083 podStartE2EDuration="6.216833057s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.600219974s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:14:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:14:45.181140366 +0000 UTC m=+1795.256909996" watchObservedRunningTime="2026-09-16 19:14:45.216833057 +0000 UTC m=+1795.292602687"
Sep 16 19:14:46.156312 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:46.155205    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch"
Sep 16 19:14:46.156312 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:46.155309    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct"
Sep 16 19:14:46.160876 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:46.160835    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct" podUID="b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:14:46.160876 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:46.160862    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch" podUID="a5b01dce-7cd8-483a-8480-8027abc6fae0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:14:47.158087 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:47.158047    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch" podUID="a5b01dce-7cd8-483a-8480-8027abc6fae0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:14:47.158644 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:47.158047    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct" podUID="b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:14:48.043737 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:48.043691    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg" podUID="b4710141-8492-43c6-a2b3-b7b48a7272d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:14:50.484466 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:50.484439    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/2.log"
Sep 16 19:14:50.488560 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:50.488538    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/2.log"
Sep 16 19:14:50.490376 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:50.490354    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovn-acl-logging/0.log"
Sep 16 19:14:50.494334 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:50.494319    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovn-acl-logging/0.log"
Sep 16 19:14:52.162655 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:52.162618    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct"
Sep 16 19:14:52.163045 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:52.162888    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch"
Sep 16 19:14:52.163277 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:52.163250    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct" podUID="b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:14:52.163568 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:52.163543    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch" podUID="a5b01dce-7cd8-483a-8480-8027abc6fae0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:14:58.044026 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:14:58.043988    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg" podUID="b4710141-8492-43c6-a2b3-b7b48a7272d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:15:02.163556 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:02.163519    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch" podUID="a5b01dce-7cd8-483a-8480-8027abc6fae0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:15:02.163973 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:02.163519    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct" podUID="b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:15:08.043609 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:08.043572    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg" podUID="b4710141-8492-43c6-a2b3-b7b48a7272d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:15:12.163321 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:12.163275    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct" podUID="b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:15:12.163810 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:12.163402    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch" podUID="a5b01dce-7cd8-483a-8480-8027abc6fae0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:15:18.044313 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:18.044278    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg" podUID="b4710141-8492-43c6-a2b3-b7b48a7272d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:15:22.164042 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:22.164002    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct" podUID="b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:15:22.164442 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:22.164002    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch" podUID="a5b01dce-7cd8-483a-8480-8027abc6fae0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:15:28.043681 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:28.043640    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg" podUID="b4710141-8492-43c6-a2b3-b7b48a7272d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:15:32.163196 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:32.163161    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct" podUID="b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:15:32.163603 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:32.163560    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch" podUID="a5b01dce-7cd8-483a-8480-8027abc6fae0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:15:38.043936 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:38.043907    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"
Sep 16 19:15:42.163401 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:42.163360    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct" podUID="b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:15:42.163819 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:42.163480    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch" podUID="a5b01dce-7cd8-483a-8480-8027abc6fae0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:15:45.306231 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:45.306202    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"]
Sep 16 19:15:45.306698 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:45.306440    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg" podUID="b4710141-8492-43c6-a2b3-b7b48a7272d6" containerName="kserve-container" containerID="cri-o://92a1ebe9a88facfb936cd54914a6935f01c492075a56c5a0fbafd537acfbddf0" gracePeriod=30
Sep 16 19:15:45.306698 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:45.306474    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg" podUID="b4710141-8492-43c6-a2b3-b7b48a7272d6" containerName="kube-rbac-proxy" containerID="cri-o://baafe2b873166c4c3e34e355e83f4fbce3cf4104fa478e8c028759ee1dc5ab99" gracePeriod=30
Sep 16 19:15:45.438192 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:45.438166    2554 generic.go:356] "Generic (PLEG): container finished" podID="b4710141-8492-43c6-a2b3-b7b48a7272d6" containerID="baafe2b873166c4c3e34e355e83f4fbce3cf4104fa478e8c028759ee1dc5ab99" exitCode=2
Sep 16 19:15:45.438310 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:45.438240    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg" event={"ID":"b4710141-8492-43c6-a2b3-b7b48a7272d6","Type":"ContainerDied","Data":"baafe2b873166c4c3e34e355e83f4fbce3cf4104fa478e8c028759ee1dc5ab99"}
Sep 16 19:15:48.039105 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:48.039058    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg" podUID="b4710141-8492-43c6-a2b3-b7b48a7272d6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Sep 16 19:15:48.043332 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:48.043301    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg" podUID="b4710141-8492-43c6-a2b3-b7b48a7272d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Sep 16 19:15:50.241248 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.241219    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"
Sep 16 19:15:50.381977 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.381890    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4710141-8492-43c6-a2b3-b7b48a7272d6-kube-rbac-proxy-sar-config\") pod \"b4710141-8492-43c6-a2b3-b7b48a7272d6\" (UID: \"b4710141-8492-43c6-a2b3-b7b48a7272d6\") "
Sep 16 19:15:50.381977 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.381948    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4710141-8492-43c6-a2b3-b7b48a7272d6-kserve-provision-location\") pod \"b4710141-8492-43c6-a2b3-b7b48a7272d6\" (UID: \"b4710141-8492-43c6-a2b3-b7b48a7272d6\") "
Sep 16 19:15:50.381977 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.381969    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4710141-8492-43c6-a2b3-b7b48a7272d6-proxy-tls\") pod \"b4710141-8492-43c6-a2b3-b7b48a7272d6\" (UID: \"b4710141-8492-43c6-a2b3-b7b48a7272d6\") "
Sep 16 19:15:50.382313 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.382011    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4710141-8492-43c6-a2b3-b7b48a7272d6-cabundle-cert\") pod \"b4710141-8492-43c6-a2b3-b7b48a7272d6\" (UID: \"b4710141-8492-43c6-a2b3-b7b48a7272d6\") "
Sep 16 19:15:50.382313 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.382052    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvfxg\" (UniqueName: \"kubernetes.io/projected/b4710141-8492-43c6-a2b3-b7b48a7272d6-kube-api-access-kvfxg\") pod \"b4710141-8492-43c6-a2b3-b7b48a7272d6\" (UID: \"b4710141-8492-43c6-a2b3-b7b48a7272d6\") "
Sep 16 19:15:50.382313 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.382272    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4710141-8492-43c6-a2b3-b7b48a7272d6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b4710141-8492-43c6-a2b3-b7b48a7272d6" (UID: "b4710141-8492-43c6-a2b3-b7b48a7272d6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:15:50.382313 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.382286    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4710141-8492-43c6-a2b3-b7b48a7272d6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b4710141-8492-43c6-a2b3-b7b48a7272d6" (UID: "b4710141-8492-43c6-a2b3-b7b48a7272d6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:15:50.382500 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.382332    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4710141-8492-43c6-a2b3-b7b48a7272d6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b4710141-8492-43c6-a2b3-b7b48a7272d6" (UID: "b4710141-8492-43c6-a2b3-b7b48a7272d6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:15:50.382500 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.382379    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b4710141-8492-43c6-a2b3-b7b48a7272d6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:15:50.382500 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.382391    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b4710141-8492-43c6-a2b3-b7b48a7272d6-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:15:50.384014 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.383992    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4710141-8492-43c6-a2b3-b7b48a7272d6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b4710141-8492-43c6-a2b3-b7b48a7272d6" (UID: "b4710141-8492-43c6-a2b3-b7b48a7272d6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:15:50.384191 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.384170    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4710141-8492-43c6-a2b3-b7b48a7272d6-kube-api-access-kvfxg" (OuterVolumeSpecName: "kube-api-access-kvfxg") pod "b4710141-8492-43c6-a2b3-b7b48a7272d6" (UID: "b4710141-8492-43c6-a2b3-b7b48a7272d6"). InnerVolumeSpecName "kube-api-access-kvfxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:15:50.470559 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.470531    2554 generic.go:356] "Generic (PLEG): container finished" podID="b4710141-8492-43c6-a2b3-b7b48a7272d6" containerID="92a1ebe9a88facfb936cd54914a6935f01c492075a56c5a0fbafd537acfbddf0" exitCode=0
Sep 16 19:15:50.470701 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.470583    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"
Sep 16 19:15:50.470701 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.470581    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg" event={"ID":"b4710141-8492-43c6-a2b3-b7b48a7272d6","Type":"ContainerDied","Data":"92a1ebe9a88facfb936cd54914a6935f01c492075a56c5a0fbafd537acfbddf0"}
Sep 16 19:15:50.470701 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.470652    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg" event={"ID":"b4710141-8492-43c6-a2b3-b7b48a7272d6","Type":"ContainerDied","Data":"cec5ef2548e10ed6a85f059151ac2451f895742f77374ccdb7268d136390d5e6"}
Sep 16 19:15:50.470701 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.470690    2554 scope.go:117] "RemoveContainer" containerID="baafe2b873166c4c3e34e355e83f4fbce3cf4104fa478e8c028759ee1dc5ab99"
Sep 16 19:15:50.481198 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.481183    2554 scope.go:117] "RemoveContainer" containerID="92a1ebe9a88facfb936cd54914a6935f01c492075a56c5a0fbafd537acfbddf0"
Sep 16 19:15:50.483492 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.483473    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b4710141-8492-43c6-a2b3-b7b48a7272d6-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:15:50.483492 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.483493    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b4710141-8492-43c6-a2b3-b7b48a7272d6-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:15:50.483608 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.483504    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvfxg\" (UniqueName: \"kubernetes.io/projected/b4710141-8492-43c6-a2b3-b7b48a7272d6-kube-api-access-kvfxg\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:15:50.488394 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.488379    2554 scope.go:117] "RemoveContainer" containerID="41de7ee37a4f765b20bfcad84ca6a396bb9577eb96e9fa98fe5b7462d1cfcd60"
Sep 16 19:15:50.496000 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.495983    2554 scope.go:117] "RemoveContainer" containerID="baafe2b873166c4c3e34e355e83f4fbce3cf4104fa478e8c028759ee1dc5ab99"
Sep 16 19:15:50.496208 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.496180    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"]
Sep 16 19:15:50.496290 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:15:50.496262    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baafe2b873166c4c3e34e355e83f4fbce3cf4104fa478e8c028759ee1dc5ab99\": container with ID starting with baafe2b873166c4c3e34e355e83f4fbce3cf4104fa478e8c028759ee1dc5ab99 not found: ID does not exist" containerID="baafe2b873166c4c3e34e355e83f4fbce3cf4104fa478e8c028759ee1dc5ab99"
Sep 16 19:15:50.496335 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.496296    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baafe2b873166c4c3e34e355e83f4fbce3cf4104fa478e8c028759ee1dc5ab99"} err="failed to get container status \"baafe2b873166c4c3e34e355e83f4fbce3cf4104fa478e8c028759ee1dc5ab99\": rpc error: code = NotFound desc = could not find container \"baafe2b873166c4c3e34e355e83f4fbce3cf4104fa478e8c028759ee1dc5ab99\": container with ID starting with baafe2b873166c4c3e34e355e83f4fbce3cf4104fa478e8c028759ee1dc5ab99 not found: ID does not exist"
Sep 16 19:15:50.496335 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.496313    2554 scope.go:117] "RemoveContainer" containerID="92a1ebe9a88facfb936cd54914a6935f01c492075a56c5a0fbafd537acfbddf0"
Sep 16 19:15:50.496558 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:15:50.496540    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92a1ebe9a88facfb936cd54914a6935f01c492075a56c5a0fbafd537acfbddf0\": container with ID starting with 92a1ebe9a88facfb936cd54914a6935f01c492075a56c5a0fbafd537acfbddf0 not found: ID does not exist" containerID="92a1ebe9a88facfb936cd54914a6935f01c492075a56c5a0fbafd537acfbddf0"
Sep 16 19:15:50.496641 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.496566    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92a1ebe9a88facfb936cd54914a6935f01c492075a56c5a0fbafd537acfbddf0"} err="failed to get container status \"92a1ebe9a88facfb936cd54914a6935f01c492075a56c5a0fbafd537acfbddf0\": rpc error: code = NotFound desc = could not find container \"92a1ebe9a88facfb936cd54914a6935f01c492075a56c5a0fbafd537acfbddf0\": container with ID starting with 92a1ebe9a88facfb936cd54914a6935f01c492075a56c5a0fbafd537acfbddf0 not found: ID does not exist"
Sep 16 19:15:50.496641 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.496588    2554 scope.go:117] "RemoveContainer" containerID="41de7ee37a4f765b20bfcad84ca6a396bb9577eb96e9fa98fe5b7462d1cfcd60"
Sep 16 19:15:50.496890 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:15:50.496875    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41de7ee37a4f765b20bfcad84ca6a396bb9577eb96e9fa98fe5b7462d1cfcd60\": container with ID starting with 41de7ee37a4f765b20bfcad84ca6a396bb9577eb96e9fa98fe5b7462d1cfcd60 not found: ID does not exist" containerID="41de7ee37a4f765b20bfcad84ca6a396bb9577eb96e9fa98fe5b7462d1cfcd60"
Sep 16 19:15:50.496938 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.496896    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41de7ee37a4f765b20bfcad84ca6a396bb9577eb96e9fa98fe5b7462d1cfcd60"} err="failed to get container status \"41de7ee37a4f765b20bfcad84ca6a396bb9577eb96e9fa98fe5b7462d1cfcd60\": rpc error: code = NotFound desc = could not find container \"41de7ee37a4f765b20bfcad84ca6a396bb9577eb96e9fa98fe5b7462d1cfcd60\": container with ID starting with 41de7ee37a4f765b20bfcad84ca6a396bb9577eb96e9fa98fe5b7462d1cfcd60 not found: ID does not exist"
Sep 16 19:15:50.501189 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.501171    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-ccf59f694-5ftbg"]
Sep 16 19:15:50.511481 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.511461    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"]
Sep 16 19:15:50.511870 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.511858    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4710141-8492-43c6-a2b3-b7b48a7272d6" containerName="kube-rbac-proxy"
Sep 16 19:15:50.511924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.511872    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4710141-8492-43c6-a2b3-b7b48a7272d6" containerName="kube-rbac-proxy"
Sep 16 19:15:50.511924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.511881    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4710141-8492-43c6-a2b3-b7b48a7272d6" containerName="storage-initializer"
Sep 16 19:15:50.511924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.511888    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4710141-8492-43c6-a2b3-b7b48a7272d6" containerName="storage-initializer"
Sep 16 19:15:50.511924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.511895    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4710141-8492-43c6-a2b3-b7b48a7272d6" containerName="kserve-container"
Sep 16 19:15:50.511924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.511901    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4710141-8492-43c6-a2b3-b7b48a7272d6" containerName="kserve-container"
Sep 16 19:15:50.512089 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.511955    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4710141-8492-43c6-a2b3-b7b48a7272d6" containerName="kserve-container"
Sep 16 19:15:50.512089 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.511962    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4710141-8492-43c6-a2b3-b7b48a7272d6" containerName="kube-rbac-proxy"
Sep 16 19:15:50.516644 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.516625    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"
Sep 16 19:15:50.518852 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.518834    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Sep 16 19:15:50.518949 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.518835    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Sep 16 19:15:50.528433 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.528416    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"]
Sep 16 19:15:50.584748 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.584709    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb3d6380-1448-4622-aabc-627adde20eb8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn\" (UID: \"eb3d6380-1448-4622-aabc-627adde20eb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"
Sep 16 19:15:50.584845 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.584767    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb3d6380-1448-4622-aabc-627adde20eb8-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn\" (UID: \"eb3d6380-1448-4622-aabc-627adde20eb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"
Sep 16 19:15:50.584900 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.584849    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb3d6380-1448-4622-aabc-627adde20eb8-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn\" (UID: \"eb3d6380-1448-4622-aabc-627adde20eb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"
Sep 16 19:15:50.584951 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.584906    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdlh8\" (UniqueName: \"kubernetes.io/projected/eb3d6380-1448-4622-aabc-627adde20eb8-kube-api-access-mdlh8\") pod \"isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn\" (UID: \"eb3d6380-1448-4622-aabc-627adde20eb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"
Sep 16 19:15:50.584951 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.584929    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb3d6380-1448-4622-aabc-627adde20eb8-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn\" (UID: \"eb3d6380-1448-4622-aabc-627adde20eb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"
Sep 16 19:15:50.685359 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.685300    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb3d6380-1448-4622-aabc-627adde20eb8-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn\" (UID: \"eb3d6380-1448-4622-aabc-627adde20eb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"
Sep 16 19:15:50.685465 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.685367    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdlh8\" (UniqueName: \"kubernetes.io/projected/eb3d6380-1448-4622-aabc-627adde20eb8-kube-api-access-mdlh8\") pod \"isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn\" (UID: \"eb3d6380-1448-4622-aabc-627adde20eb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"
Sep 16 19:15:50.685465 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.685399    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb3d6380-1448-4622-aabc-627adde20eb8-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn\" (UID: \"eb3d6380-1448-4622-aabc-627adde20eb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"
Sep 16 19:15:50.685583 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.685470    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb3d6380-1448-4622-aabc-627adde20eb8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn\" (UID: \"eb3d6380-1448-4622-aabc-627adde20eb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"
Sep 16 19:15:50.685583 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.685511    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb3d6380-1448-4622-aabc-627adde20eb8-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn\" (UID: \"eb3d6380-1448-4622-aabc-627adde20eb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"
Sep 16 19:15:50.685671 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:15:50.685644    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-serving-cert: secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Sep 16 19:15:50.685748 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:15:50.685731    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb3d6380-1448-4622-aabc-627adde20eb8-proxy-tls podName:eb3d6380-1448-4622-aabc-627adde20eb8 nodeName:}" failed. No retries permitted until 2026-09-16 19:15:51.18569935 +0000 UTC m=+1861.261468980 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/eb3d6380-1448-4622-aabc-627adde20eb8-proxy-tls") pod "isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn" (UID: "eb3d6380-1448-4622-aabc-627adde20eb8") : secret "isvc-sklearn-v2-mixed-predictor-serving-cert" not found
Sep 16 19:15:50.685848 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.685833    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb3d6380-1448-4622-aabc-627adde20eb8-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn\" (UID: \"eb3d6380-1448-4622-aabc-627adde20eb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"
Sep 16 19:15:50.686018 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.686000    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb3d6380-1448-4622-aabc-627adde20eb8-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn\" (UID: \"eb3d6380-1448-4622-aabc-627adde20eb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"
Sep 16 19:15:50.686099 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.686082    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb3d6380-1448-4622-aabc-627adde20eb8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn\" (UID: \"eb3d6380-1448-4622-aabc-627adde20eb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"
Sep 16 19:15:50.696058 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:50.696039    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdlh8\" (UniqueName: \"kubernetes.io/projected/eb3d6380-1448-4622-aabc-627adde20eb8-kube-api-access-mdlh8\") pod \"isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn\" (UID: \"eb3d6380-1448-4622-aabc-627adde20eb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"
Sep 16 19:15:51.189065 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:51.189032    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb3d6380-1448-4622-aabc-627adde20eb8-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn\" (UID: \"eb3d6380-1448-4622-aabc-627adde20eb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"
Sep 16 19:15:51.191290 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:51.191271    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb3d6380-1448-4622-aabc-627adde20eb8-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn\" (UID: \"eb3d6380-1448-4622-aabc-627adde20eb8\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"
Sep 16 19:15:51.426393 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:51.426364    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"
Sep 16 19:15:51.706635 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:51.706605    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"]
Sep 16 19:15:51.708299 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:15:51.708274    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb3d6380_1448_4622_aabc_627adde20eb8.slice/crio-7e552a9d3eb29a29785eb1151e63bc25cfc991a4f63fb859ee6a2b4792c8da07 WatchSource:0}: Error finding container 7e552a9d3eb29a29785eb1151e63bc25cfc991a4f63fb859ee6a2b4792c8da07: Status 404 returned error can't find the container with id 7e552a9d3eb29a29785eb1151e63bc25cfc991a4f63fb859ee6a2b4792c8da07
Sep 16 19:15:52.163198 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:52.163161    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct" podUID="b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:15:52.163565 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:52.163526    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch" podUID="a5b01dce-7cd8-483a-8480-8027abc6fae0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Sep 16 19:15:52.443953 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:52.443871    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4710141-8492-43c6-a2b3-b7b48a7272d6" path="/var/lib/kubelet/pods/b4710141-8492-43c6-a2b3-b7b48a7272d6/volumes"
Sep 16 19:15:52.485397 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:52.485367    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn" event={"ID":"eb3d6380-1448-4622-aabc-627adde20eb8","Type":"ContainerStarted","Data":"3a590ff3efd754277f284699eb3daa7f3b1b08c238544df49c7eca86126dbdf8"}
Sep 16 19:15:52.485531 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:52.485419    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn" event={"ID":"eb3d6380-1448-4622-aabc-627adde20eb8","Type":"ContainerStarted","Data":"7e552a9d3eb29a29785eb1151e63bc25cfc991a4f63fb859ee6a2b4792c8da07"}
Sep 16 19:15:53.438897 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:53.438860    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct" podUID="b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Sep 16 19:15:55.502178 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:55.502101    2554 generic.go:356] "Generic (PLEG): container finished" podID="eb3d6380-1448-4622-aabc-627adde20eb8" containerID="3a590ff3efd754277f284699eb3daa7f3b1b08c238544df49c7eca86126dbdf8" exitCode=0
Sep 16 19:15:55.502178 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:55.502173    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn" event={"ID":"eb3d6380-1448-4622-aabc-627adde20eb8","Type":"ContainerDied","Data":"3a590ff3efd754277f284699eb3daa7f3b1b08c238544df49c7eca86126dbdf8"}
Sep 16 19:15:56.508649 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:56.508620    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn" event={"ID":"eb3d6380-1448-4622-aabc-627adde20eb8","Type":"ContainerStarted","Data":"4f23ff785de6fcfddfd104c58206302623e852d250abd92870e13c5585217efd"}
Sep 16 19:15:56.509057 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:56.508676    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn" event={"ID":"eb3d6380-1448-4622-aabc-627adde20eb8","Type":"ContainerStarted","Data":"89cd54984a31b302b1c077fc6f8ff15c9085ebdb63539cb40f9ef95228330337"}
Sep 16 19:15:56.509057 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:56.508863    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"
Sep 16 19:15:56.532156 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:56.532114    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn" podStartSLOduration=2.847423922 podStartE2EDuration="6.532102914s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.684678992s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:15:56.530100332 +0000 UTC m=+1866.605869986" watchObservedRunningTime="2026-09-16 19:15:56.532102914 +0000 UTC m=+1866.607872545"
Sep 16 19:15:57.513571 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:57.513541    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"
Sep 16 19:15:57.514641 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:57.514613    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn" podUID="eb3d6380-1448-4622-aabc-627adde20eb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:15:58.443425 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:58.443396    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch"
Sep 16 19:15:58.518000 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:58.517961    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn" podUID="eb3d6380-1448-4622-aabc-627adde20eb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:15:58.595895 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:58.595865    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct"]
Sep 16 19:15:58.596079 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:58.596046    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct" podUID="b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" containerName="kserve-container" containerID="cri-o://e30958675e511eea6934a04123e1157c817e814c56601801c410c24c076b4e62" gracePeriod=30
Sep 16 19:15:58.596132 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:58.596111    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct" podUID="b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" containerName="kube-rbac-proxy" containerID="cri-o://de664d5f230ccdf4b3c876ec9dd619db6f79da81c801690fc2a1354bd121b70e" gracePeriod=30
Sep 16 19:15:58.596887 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:58.596870    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct"
Sep 16 19:15:59.525219 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:59.525186    2554 generic.go:356] "Generic (PLEG): container finished" podID="b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" containerID="de664d5f230ccdf4b3c876ec9dd619db6f79da81c801690fc2a1354bd121b70e" exitCode=2
Sep 16 19:15:59.525585 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:15:59.525258    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct" event={"ID":"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a","Type":"ContainerDied","Data":"de664d5f230ccdf4b3c876ec9dd619db6f79da81c801690fc2a1354bd121b70e"}
Sep 16 19:16:02.158641 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.158596    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct" podUID="b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Sep 16 19:16:02.435940 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.435916    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct"
Sep 16 19:16:02.541574 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.541540    2554 generic.go:356] "Generic (PLEG): container finished" podID="b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" containerID="e30958675e511eea6934a04123e1157c817e814c56601801c410c24c076b4e62" exitCode=0
Sep 16 19:16:02.541706 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.541595    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct"
Sep 16 19:16:02.541706 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.541619    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct" event={"ID":"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a","Type":"ContainerDied","Data":"e30958675e511eea6934a04123e1157c817e814c56601801c410c24c076b4e62"}
Sep 16 19:16:02.541706 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.541668    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct" event={"ID":"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a","Type":"ContainerDied","Data":"1def290ef9f4ee2972b9b4a5981267450774e439f97bf61a4218a07689521476"}
Sep 16 19:16:02.541706 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.541694    2554 scope.go:117] "RemoveContainer" containerID="de664d5f230ccdf4b3c876ec9dd619db6f79da81c801690fc2a1354bd121b70e"
Sep 16 19:16:02.556849 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.556830    2554 scope.go:117] "RemoveContainer" containerID="e30958675e511eea6934a04123e1157c817e814c56601801c410c24c076b4e62"
Sep 16 19:16:02.564345 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.564329    2554 scope.go:117] "RemoveContainer" containerID="0489d0dd28f1f1b3fbe99cbfb98e767dbe59186437e722d944dba8235453e70b"
Sep 16 19:16:02.571197 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.571179    2554 scope.go:117] "RemoveContainer" containerID="de664d5f230ccdf4b3c876ec9dd619db6f79da81c801690fc2a1354bd121b70e"
Sep 16 19:16:02.571434 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:16:02.571417    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de664d5f230ccdf4b3c876ec9dd619db6f79da81c801690fc2a1354bd121b70e\": container with ID starting with de664d5f230ccdf4b3c876ec9dd619db6f79da81c801690fc2a1354bd121b70e not found: ID does not exist" containerID="de664d5f230ccdf4b3c876ec9dd619db6f79da81c801690fc2a1354bd121b70e"
Sep 16 19:16:02.571479 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.571441    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de664d5f230ccdf4b3c876ec9dd619db6f79da81c801690fc2a1354bd121b70e"} err="failed to get container status \"de664d5f230ccdf4b3c876ec9dd619db6f79da81c801690fc2a1354bd121b70e\": rpc error: code = NotFound desc = could not find container \"de664d5f230ccdf4b3c876ec9dd619db6f79da81c801690fc2a1354bd121b70e\": container with ID starting with de664d5f230ccdf4b3c876ec9dd619db6f79da81c801690fc2a1354bd121b70e not found: ID does not exist"
Sep 16 19:16:02.571479 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.571457    2554 scope.go:117] "RemoveContainer" containerID="e30958675e511eea6934a04123e1157c817e814c56601801c410c24c076b4e62"
Sep 16 19:16:02.571708 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:16:02.571689    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e30958675e511eea6934a04123e1157c817e814c56601801c410c24c076b4e62\": container with ID starting with e30958675e511eea6934a04123e1157c817e814c56601801c410c24c076b4e62 not found: ID does not exist" containerID="e30958675e511eea6934a04123e1157c817e814c56601801c410c24c076b4e62"
Sep 16 19:16:02.571772 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.571728    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e30958675e511eea6934a04123e1157c817e814c56601801c410c24c076b4e62"} err="failed to get container status \"e30958675e511eea6934a04123e1157c817e814c56601801c410c24c076b4e62\": rpc error: code = NotFound desc = could not find container \"e30958675e511eea6934a04123e1157c817e814c56601801c410c24c076b4e62\": container with ID starting with e30958675e511eea6934a04123e1157c817e814c56601801c410c24c076b4e62 not found: ID does not exist"
Sep 16 19:16:02.571772 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.571746    2554 scope.go:117] "RemoveContainer" containerID="0489d0dd28f1f1b3fbe99cbfb98e767dbe59186437e722d944dba8235453e70b"
Sep 16 19:16:02.571965 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:16:02.571951    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0489d0dd28f1f1b3fbe99cbfb98e767dbe59186437e722d944dba8235453e70b\": container with ID starting with 0489d0dd28f1f1b3fbe99cbfb98e767dbe59186437e722d944dba8235453e70b not found: ID does not exist" containerID="0489d0dd28f1f1b3fbe99cbfb98e767dbe59186437e722d944dba8235453e70b"
Sep 16 19:16:02.572008 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.571968    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0489d0dd28f1f1b3fbe99cbfb98e767dbe59186437e722d944dba8235453e70b"} err="failed to get container status \"0489d0dd28f1f1b3fbe99cbfb98e767dbe59186437e722d944dba8235453e70b\": rpc error: code = NotFound desc = could not find container \"0489d0dd28f1f1b3fbe99cbfb98e767dbe59186437e722d944dba8235453e70b\": container with ID starting with 0489d0dd28f1f1b3fbe99cbfb98e767dbe59186437e722d944dba8235453e70b not found: ID does not exist"
Sep 16 19:16:02.584228 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.584211    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-cabundle-cert\") pod \"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a\" (UID: \"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a\") "
Sep 16 19:16:02.584293 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.584244    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgsbt\" (UniqueName: \"kubernetes.io/projected/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-kube-api-access-cgsbt\") pod \"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a\" (UID: \"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a\") "
Sep 16 19:16:02.584293 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.584287    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-proxy-tls\") pod \"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a\" (UID: \"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a\") "
Sep 16 19:16:02.584380 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.584320    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-kube-rbac-proxy-sar-config\") pod \"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a\" (UID: \"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a\") "
Sep 16 19:16:02.584380 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.584338    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-kserve-provision-location\") pod \"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a\" (UID: \"b6b3246b-d1cb-429a-9cc7-f29ea2ed127a\") "
Sep 16 19:16:02.584517 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.584497    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" (UID: "b6b3246b-d1cb-429a-9cc7-f29ea2ed127a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:16:02.584705 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.584681    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" (UID: "b6b3246b-d1cb-429a-9cc7-f29ea2ed127a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:02.584705 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.584685    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" (UID: "b6b3246b-d1cb-429a-9cc7-f29ea2ed127a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:16:02.586225 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.586201    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-kube-api-access-cgsbt" (OuterVolumeSpecName: "kube-api-access-cgsbt") pod "b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" (UID: "b6b3246b-d1cb-429a-9cc7-f29ea2ed127a"). InnerVolumeSpecName "kube-api-access-cgsbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:16:02.586287 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.586207    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" (UID: "b6b3246b-d1cb-429a-9cc7-f29ea2ed127a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:16:02.685775 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.685711    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:16:02.685775 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.685744    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:16:02.685775 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.685754    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:16:02.685775 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.685763    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgsbt\" (UniqueName: \"kubernetes.io/projected/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-kube-api-access-cgsbt\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:16:02.685775 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.685772    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:16:02.864612 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.864588    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct"]
Sep 16 19:16:02.870363 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:02.870343    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-6fb974599-bpsct"]
Sep 16 19:16:03.522140 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:03.522115    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"
Sep 16 19:16:03.522597 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:03.522570    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn" podUID="eb3d6380-1448-4622-aabc-627adde20eb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:16:04.443248 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:04.443208    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" path="/var/lib/kubelet/pods/b6b3246b-d1cb-429a-9cc7-f29ea2ed127a/volumes"
Sep 16 19:16:13.523448 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:13.523368    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn" podUID="eb3d6380-1448-4622-aabc-627adde20eb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:16:19.669048 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:19.669014    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch"]
Sep 16 19:16:19.669633 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:19.669240    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch" podUID="a5b01dce-7cd8-483a-8480-8027abc6fae0" containerName="kserve-container" containerID="cri-o://6195e4a54c6fe328d536b4881e611e020feaf1480bbbf7a88b97a477c16b2997" gracePeriod=30
Sep 16 19:16:19.669633 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:19.669321    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch" podUID="a5b01dce-7cd8-483a-8480-8027abc6fae0" containerName="kube-rbac-proxy" containerID="cri-o://477e5855aeb975a0c92bd670bb859956bef5eb8c29582988fd280fe0b732ede3" gracePeriod=30
Sep 16 19:16:20.631385 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:20.631356    2554 generic.go:356] "Generic (PLEG): container finished" podID="a5b01dce-7cd8-483a-8480-8027abc6fae0" containerID="477e5855aeb975a0c92bd670bb859956bef5eb8c29582988fd280fe0b732ede3" exitCode=2
Sep 16 19:16:20.631549 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:20.631428    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch" event={"ID":"a5b01dce-7cd8-483a-8480-8027abc6fae0","Type":"ContainerDied","Data":"477e5855aeb975a0c92bd670bb859956bef5eb8c29582988fd280fe0b732ede3"}
Sep 16 19:16:22.159058 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:22.159019    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch" podUID="a5b01dce-7cd8-483a-8480-8027abc6fae0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Sep 16 19:16:23.522981 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:23.522940    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn" podUID="eb3d6380-1448-4622-aabc-627adde20eb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:16:24.113072 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.113042    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch"
Sep 16 19:16:24.151669 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.151643    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5b01dce-7cd8-483a-8480-8027abc6fae0-proxy-tls\") pod \"a5b01dce-7cd8-483a-8480-8027abc6fae0\" (UID: \"a5b01dce-7cd8-483a-8480-8027abc6fae0\") "
Sep 16 19:16:24.151837 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.151706    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5b01dce-7cd8-483a-8480-8027abc6fae0-cabundle-cert\") pod \"a5b01dce-7cd8-483a-8480-8027abc6fae0\" (UID: \"a5b01dce-7cd8-483a-8480-8027abc6fae0\") "
Sep 16 19:16:24.151837 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.151778    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5b01dce-7cd8-483a-8480-8027abc6fae0-kserve-provision-location\") pod \"a5b01dce-7cd8-483a-8480-8027abc6fae0\" (UID: \"a5b01dce-7cd8-483a-8480-8027abc6fae0\") "
Sep 16 19:16:24.151837 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.151808    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5b01dce-7cd8-483a-8480-8027abc6fae0-kube-rbac-proxy-sar-config\") pod \"a5b01dce-7cd8-483a-8480-8027abc6fae0\" (UID: \"a5b01dce-7cd8-483a-8480-8027abc6fae0\") "
Sep 16 19:16:24.152016 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.151842    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89vcx\" (UniqueName: \"kubernetes.io/projected/a5b01dce-7cd8-483a-8480-8027abc6fae0-kube-api-access-89vcx\") pod \"a5b01dce-7cd8-483a-8480-8027abc6fae0\" (UID: \"a5b01dce-7cd8-483a-8480-8027abc6fae0\") "
Sep 16 19:16:24.152170 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.152139    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b01dce-7cd8-483a-8480-8027abc6fae0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a5b01dce-7cd8-483a-8480-8027abc6fae0" (UID: "a5b01dce-7cd8-483a-8480-8027abc6fae0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:16:24.152283 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.152184    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5b01dce-7cd8-483a-8480-8027abc6fae0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a5b01dce-7cd8-483a-8480-8027abc6fae0" (UID: "a5b01dce-7cd8-483a-8480-8027abc6fae0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:16:24.152337 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.152310    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5b01dce-7cd8-483a-8480-8027abc6fae0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a5b01dce-7cd8-483a-8480-8027abc6fae0" (UID: "a5b01dce-7cd8-483a-8480-8027abc6fae0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:16:24.153922 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.153897    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5b01dce-7cd8-483a-8480-8027abc6fae0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a5b01dce-7cd8-483a-8480-8027abc6fae0" (UID: "a5b01dce-7cd8-483a-8480-8027abc6fae0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:16:24.154024 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.154008    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b01dce-7cd8-483a-8480-8027abc6fae0-kube-api-access-89vcx" (OuterVolumeSpecName: "kube-api-access-89vcx") pod "a5b01dce-7cd8-483a-8480-8027abc6fae0" (UID: "a5b01dce-7cd8-483a-8480-8027abc6fae0"). InnerVolumeSpecName "kube-api-access-89vcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:16:24.252733 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.252694    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a5b01dce-7cd8-483a-8480-8027abc6fae0-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:16:24.252879 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.252740    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a5b01dce-7cd8-483a-8480-8027abc6fae0-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:16:24.252879 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.252751    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a5b01dce-7cd8-483a-8480-8027abc6fae0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:16:24.252879 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.252761    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89vcx\" (UniqueName: \"kubernetes.io/projected/a5b01dce-7cd8-483a-8480-8027abc6fae0-kube-api-access-89vcx\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:16:24.252879 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.252771    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a5b01dce-7cd8-483a-8480-8027abc6fae0-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:16:24.653147 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.653050    2554 generic.go:356] "Generic (PLEG): container finished" podID="a5b01dce-7cd8-483a-8480-8027abc6fae0" containerID="6195e4a54c6fe328d536b4881e611e020feaf1480bbbf7a88b97a477c16b2997" exitCode=0
Sep 16 19:16:24.653147 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.653077    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch" event={"ID":"a5b01dce-7cd8-483a-8480-8027abc6fae0","Type":"ContainerDied","Data":"6195e4a54c6fe328d536b4881e611e020feaf1480bbbf7a88b97a477c16b2997"}
Sep 16 19:16:24.653147 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.653109    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch"
Sep 16 19:16:24.653147 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.653141    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch" event={"ID":"a5b01dce-7cd8-483a-8480-8027abc6fae0","Type":"ContainerDied","Data":"2d8df22d09ffb64d2874a4eae9097e7e40d150305ee1eb4e0fd433dd1e0c2fb5"}
Sep 16 19:16:24.653838 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.653170    2554 scope.go:117] "RemoveContainer" containerID="477e5855aeb975a0c92bd670bb859956bef5eb8c29582988fd280fe0b732ede3"
Sep 16 19:16:24.661754 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.661735    2554 scope.go:117] "RemoveContainer" containerID="6195e4a54c6fe328d536b4881e611e020feaf1480bbbf7a88b97a477c16b2997"
Sep 16 19:16:24.669317 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.669297    2554 scope.go:117] "RemoveContainer" containerID="2d2f0eed534cb46d12a306459a29641c9113a9c0ce198c646f3f6ca1b2e5ed57"
Sep 16 19:16:24.672389 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.672366    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch"]
Sep 16 19:16:24.677513 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.677486    2554 scope.go:117] "RemoveContainer" containerID="477e5855aeb975a0c92bd670bb859956bef5eb8c29582988fd280fe0b732ede3"
Sep 16 19:16:24.677810 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:16:24.677789    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"477e5855aeb975a0c92bd670bb859956bef5eb8c29582988fd280fe0b732ede3\": container with ID starting with 477e5855aeb975a0c92bd670bb859956bef5eb8c29582988fd280fe0b732ede3 not found: ID does not exist" containerID="477e5855aeb975a0c92bd670bb859956bef5eb8c29582988fd280fe0b732ede3"
Sep 16 19:16:24.677909 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.677816    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"477e5855aeb975a0c92bd670bb859956bef5eb8c29582988fd280fe0b732ede3"} err="failed to get container status \"477e5855aeb975a0c92bd670bb859956bef5eb8c29582988fd280fe0b732ede3\": rpc error: code = NotFound desc = could not find container \"477e5855aeb975a0c92bd670bb859956bef5eb8c29582988fd280fe0b732ede3\": container with ID starting with 477e5855aeb975a0c92bd670bb859956bef5eb8c29582988fd280fe0b732ede3 not found: ID does not exist"
Sep 16 19:16:24.677909 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.677833    2554 scope.go:117] "RemoveContainer" containerID="6195e4a54c6fe328d536b4881e611e020feaf1480bbbf7a88b97a477c16b2997"
Sep 16 19:16:24.678046 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.677927    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-847844f798-qldch"]
Sep 16 19:16:24.678110 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:16:24.678092    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6195e4a54c6fe328d536b4881e611e020feaf1480bbbf7a88b97a477c16b2997\": container with ID starting with 6195e4a54c6fe328d536b4881e611e020feaf1480bbbf7a88b97a477c16b2997 not found: ID does not exist" containerID="6195e4a54c6fe328d536b4881e611e020feaf1480bbbf7a88b97a477c16b2997"
Sep 16 19:16:24.678148 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.678115    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6195e4a54c6fe328d536b4881e611e020feaf1480bbbf7a88b97a477c16b2997"} err="failed to get container status \"6195e4a54c6fe328d536b4881e611e020feaf1480bbbf7a88b97a477c16b2997\": rpc error: code = NotFound desc = could not find container \"6195e4a54c6fe328d536b4881e611e020feaf1480bbbf7a88b97a477c16b2997\": container with ID starting with 6195e4a54c6fe328d536b4881e611e020feaf1480bbbf7a88b97a477c16b2997 not found: ID does not exist"
Sep 16 19:16:24.678148 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.678127    2554 scope.go:117] "RemoveContainer" containerID="2d2f0eed534cb46d12a306459a29641c9113a9c0ce198c646f3f6ca1b2e5ed57"
Sep 16 19:16:24.678337 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:16:24.678321    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d2f0eed534cb46d12a306459a29641c9113a9c0ce198c646f3f6ca1b2e5ed57\": container with ID starting with 2d2f0eed534cb46d12a306459a29641c9113a9c0ce198c646f3f6ca1b2e5ed57 not found: ID does not exist" containerID="2d2f0eed534cb46d12a306459a29641c9113a9c0ce198c646f3f6ca1b2e5ed57"
Sep 16 19:16:24.678381 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.678342    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d2f0eed534cb46d12a306459a29641c9113a9c0ce198c646f3f6ca1b2e5ed57"} err="failed to get container status \"2d2f0eed534cb46d12a306459a29641c9113a9c0ce198c646f3f6ca1b2e5ed57\": rpc error: code = NotFound desc = could not find container \"2d2f0eed534cb46d12a306459a29641c9113a9c0ce198c646f3f6ca1b2e5ed57\": container with ID starting with 2d2f0eed534cb46d12a306459a29641c9113a9c0ce198c646f3f6ca1b2e5ed57 not found: ID does not exist"
Sep 16 19:16:24.932954 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.932880    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"]
Sep 16 19:16:24.933225 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.933213    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" containerName="kube-rbac-proxy"
Sep 16 19:16:24.933269 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.933227    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" containerName="kube-rbac-proxy"
Sep 16 19:16:24.933269 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.933243    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" containerName="kserve-container"
Sep 16 19:16:24.933269 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.933249    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" containerName="kserve-container"
Sep 16 19:16:24.933269 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.933260    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5b01dce-7cd8-483a-8480-8027abc6fae0" containerName="kube-rbac-proxy"
Sep 16 19:16:24.933269 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.933266    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5b01dce-7cd8-483a-8480-8027abc6fae0" containerName="kube-rbac-proxy"
Sep 16 19:16:24.933423 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.933278    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5b01dce-7cd8-483a-8480-8027abc6fae0" containerName="storage-initializer"
Sep 16 19:16:24.933423 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.933283    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5b01dce-7cd8-483a-8480-8027abc6fae0" containerName="storage-initializer"
Sep 16 19:16:24.933423 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.933291    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" containerName="storage-initializer"
Sep 16 19:16:24.933423 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.933296    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" containerName="storage-initializer"
Sep 16 19:16:24.933423 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.933303    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5b01dce-7cd8-483a-8480-8027abc6fae0" containerName="kserve-container"
Sep 16 19:16:24.933423 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.933309    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5b01dce-7cd8-483a-8480-8027abc6fae0" containerName="kserve-container"
Sep 16 19:16:24.933423 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.933358    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5b01dce-7cd8-483a-8480-8027abc6fae0" containerName="kserve-container"
Sep 16 19:16:24.933423 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.933367    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" containerName="kube-rbac-proxy"
Sep 16 19:16:24.933423 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.933376    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6b3246b-d1cb-429a-9cc7-f29ea2ed127a" containerName="kserve-container"
Sep 16 19:16:24.933423 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.933382    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5b01dce-7cd8-483a-8480-8027abc6fae0" containerName="kube-rbac-proxy"
Sep 16 19:16:24.937771 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.937752    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"
Sep 16 19:16:24.940297 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.940280    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 16 19:16:24.940608 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.940594    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 16 19:16:24.950592 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.950569    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"]
Sep 16 19:16:24.958249 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.958221    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqhfb\" (UniqueName: \"kubernetes.io/projected/05e376de-c2b1-4b26-9d67-2b9fa6be5752-kube-api-access-sqhfb\") pod \"isvc-canary-promote-predictor-649dd98487-bfvz6\" (UID: \"05e376de-c2b1-4b26-9d67-2b9fa6be5752\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"
Sep 16 19:16:24.958357 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.958309    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05e376de-c2b1-4b26-9d67-2b9fa6be5752-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-649dd98487-bfvz6\" (UID: \"05e376de-c2b1-4b26-9d67-2b9fa6be5752\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"
Sep 16 19:16:24.958436 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.958358    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05e376de-c2b1-4b26-9d67-2b9fa6be5752-kserve-provision-location\") pod \"isvc-canary-promote-predictor-649dd98487-bfvz6\" (UID: \"05e376de-c2b1-4b26-9d67-2b9fa6be5752\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"
Sep 16 19:16:24.958436 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.958381    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05e376de-c2b1-4b26-9d67-2b9fa6be5752-proxy-tls\") pod \"isvc-canary-promote-predictor-649dd98487-bfvz6\" (UID: \"05e376de-c2b1-4b26-9d67-2b9fa6be5752\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"
Sep 16 19:16:24.958436 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:24.958402    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05e376de-c2b1-4b26-9d67-2b9fa6be5752-cabundle-cert\") pod \"isvc-canary-promote-predictor-649dd98487-bfvz6\" (UID: \"05e376de-c2b1-4b26-9d67-2b9fa6be5752\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"
Sep 16 19:16:25.059518 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:25.059474    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqhfb\" (UniqueName: \"kubernetes.io/projected/05e376de-c2b1-4b26-9d67-2b9fa6be5752-kube-api-access-sqhfb\") pod \"isvc-canary-promote-predictor-649dd98487-bfvz6\" (UID: \"05e376de-c2b1-4b26-9d67-2b9fa6be5752\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"
Sep 16 19:16:25.059714 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:25.059531    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05e376de-c2b1-4b26-9d67-2b9fa6be5752-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-649dd98487-bfvz6\" (UID: \"05e376de-c2b1-4b26-9d67-2b9fa6be5752\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"
Sep 16 19:16:25.059714 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:25.059580    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05e376de-c2b1-4b26-9d67-2b9fa6be5752-kserve-provision-location\") pod \"isvc-canary-promote-predictor-649dd98487-bfvz6\" (UID: \"05e376de-c2b1-4b26-9d67-2b9fa6be5752\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"
Sep 16 19:16:25.059714 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:25.059606    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05e376de-c2b1-4b26-9d67-2b9fa6be5752-proxy-tls\") pod \"isvc-canary-promote-predictor-649dd98487-bfvz6\" (UID: \"05e376de-c2b1-4b26-9d67-2b9fa6be5752\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"
Sep 16 19:16:25.059714 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:25.059662    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05e376de-c2b1-4b26-9d67-2b9fa6be5752-cabundle-cert\") pod \"isvc-canary-promote-predictor-649dd98487-bfvz6\" (UID: \"05e376de-c2b1-4b26-9d67-2b9fa6be5752\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"
Sep 16 19:16:25.059714 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:16:25.059691    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 16 19:16:25.059988 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:16:25.059779    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05e376de-c2b1-4b26-9d67-2b9fa6be5752-proxy-tls podName:05e376de-c2b1-4b26-9d67-2b9fa6be5752 nodeName:}" failed. No retries permitted until 2026-09-16 19:16:25.559758329 +0000 UTC m=+1895.635527939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/05e376de-c2b1-4b26-9d67-2b9fa6be5752-proxy-tls") pod "isvc-canary-promote-predictor-649dd98487-bfvz6" (UID: "05e376de-c2b1-4b26-9d67-2b9fa6be5752") : secret "isvc-canary-promote-predictor-serving-cert" not found
Sep 16 19:16:25.060072 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:25.060043    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05e376de-c2b1-4b26-9d67-2b9fa6be5752-kserve-provision-location\") pod \"isvc-canary-promote-predictor-649dd98487-bfvz6\" (UID: \"05e376de-c2b1-4b26-9d67-2b9fa6be5752\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"
Sep 16 19:16:25.060353 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:25.060334    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05e376de-c2b1-4b26-9d67-2b9fa6be5752-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-649dd98487-bfvz6\" (UID: \"05e376de-c2b1-4b26-9d67-2b9fa6be5752\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"
Sep 16 19:16:25.060353 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:25.060347    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05e376de-c2b1-4b26-9d67-2b9fa6be5752-cabundle-cert\") pod \"isvc-canary-promote-predictor-649dd98487-bfvz6\" (UID: \"05e376de-c2b1-4b26-9d67-2b9fa6be5752\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"
Sep 16 19:16:25.068581 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:25.068562    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqhfb\" (UniqueName: \"kubernetes.io/projected/05e376de-c2b1-4b26-9d67-2b9fa6be5752-kube-api-access-sqhfb\") pod \"isvc-canary-promote-predictor-649dd98487-bfvz6\" (UID: \"05e376de-c2b1-4b26-9d67-2b9fa6be5752\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"
Sep 16 19:16:25.563903 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:25.563869    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05e376de-c2b1-4b26-9d67-2b9fa6be5752-proxy-tls\") pod \"isvc-canary-promote-predictor-649dd98487-bfvz6\" (UID: \"05e376de-c2b1-4b26-9d67-2b9fa6be5752\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"
Sep 16 19:16:25.566261 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:25.566239    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05e376de-c2b1-4b26-9d67-2b9fa6be5752-proxy-tls\") pod \"isvc-canary-promote-predictor-649dd98487-bfvz6\" (UID: \"05e376de-c2b1-4b26-9d67-2b9fa6be5752\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"
Sep 16 19:16:25.848012 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:25.847927    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"
Sep 16 19:16:26.122672 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:26.122602    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"]
Sep 16 19:16:26.124778 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:16:26.124754    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e376de_c2b1_4b26_9d67_2b9fa6be5752.slice/crio-1bb1ea4ea2dac0d8deabd6a8f301604288d108285d065f6451e9990f90ae1c10 WatchSource:0}: Error finding container 1bb1ea4ea2dac0d8deabd6a8f301604288d108285d065f6451e9990f90ae1c10: Status 404 returned error can't find the container with id 1bb1ea4ea2dac0d8deabd6a8f301604288d108285d065f6451e9990f90ae1c10
Sep 16 19:16:26.443942 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:26.443863    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5b01dce-7cd8-483a-8480-8027abc6fae0" path="/var/lib/kubelet/pods/a5b01dce-7cd8-483a-8480-8027abc6fae0/volumes"
Sep 16 19:16:26.666728 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:26.666694    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6" event={"ID":"05e376de-c2b1-4b26-9d67-2b9fa6be5752","Type":"ContainerStarted","Data":"ce581ff56e24cc959976b36410f4767a1f5c943b7012c9041cb0fab9ecdeac6f"}
Sep 16 19:16:26.666901 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:26.666774    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6" event={"ID":"05e376de-c2b1-4b26-9d67-2b9fa6be5752","Type":"ContainerStarted","Data":"1bb1ea4ea2dac0d8deabd6a8f301604288d108285d065f6451e9990f90ae1c10"}
Sep 16 19:16:30.687129 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:30.687092    2554 generic.go:356] "Generic (PLEG): container finished" podID="05e376de-c2b1-4b26-9d67-2b9fa6be5752" containerID="ce581ff56e24cc959976b36410f4767a1f5c943b7012c9041cb0fab9ecdeac6f" exitCode=0
Sep 16 19:16:30.689276 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:30.687167    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6" event={"ID":"05e376de-c2b1-4b26-9d67-2b9fa6be5752","Type":"ContainerDied","Data":"ce581ff56e24cc959976b36410f4767a1f5c943b7012c9041cb0fab9ecdeac6f"}
Sep 16 19:16:31.694963 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:31.694927    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6" event={"ID":"05e376de-c2b1-4b26-9d67-2b9fa6be5752","Type":"ContainerStarted","Data":"f5c56ed7b32fbbc2c673eb0acc59d45a6a7e5f866568a8b04cbe8239c29d1eb4"}
Sep 16 19:16:31.695358 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:31.694995    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6" event={"ID":"05e376de-c2b1-4b26-9d67-2b9fa6be5752","Type":"ContainerStarted","Data":"552c4c1da507db2676f264753c12e069514faee0b32f9f798edfdeca12584f5f"}
Sep 16 19:16:31.695358 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:31.695175    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"
Sep 16 19:16:31.715545 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:31.715502    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6" podStartSLOduration=3.921863939 podStartE2EDuration="7.715488816s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.793624877s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:16:31.713631325 +0000 UTC m=+1901.789400956" watchObservedRunningTime="2026-09-16 19:16:31.715488816 +0000 UTC m=+1901.791258448"
Sep 16 19:16:32.700087 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:32.700057    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"
Sep 16 19:16:32.701441 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:32.701413    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6" podUID="05e376de-c2b1-4b26-9d67-2b9fa6be5752" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:16:33.523029 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:33.522990    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn" podUID="eb3d6380-1448-4622-aabc-627adde20eb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:16:33.704677 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:33.704640    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6" podUID="05e376de-c2b1-4b26-9d67-2b9fa6be5752" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:16:38.709572 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:38.709543    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"
Sep 16 19:16:38.710136 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:38.710107    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6" podUID="05e376de-c2b1-4b26-9d67-2b9fa6be5752" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:16:43.522968 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:43.522929    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn" podUID="eb3d6380-1448-4622-aabc-627adde20eb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:16:48.710643 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:48.710600    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6" podUID="05e376de-c2b1-4b26-9d67-2b9fa6be5752" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:16:53.522646 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:53.522606    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn" podUID="eb3d6380-1448-4622-aabc-627adde20eb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:16:58.710847 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:16:58.710810    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6" podUID="05e376de-c2b1-4b26-9d67-2b9fa6be5752" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:17:03.522856 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:03.522820    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn" podUID="eb3d6380-1448-4622-aabc-627adde20eb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:17:08.710765 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:08.710728    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6" podUID="05e376de-c2b1-4b26-9d67-2b9fa6be5752" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:17:13.528164 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:13.528135    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"
Sep 16 19:17:18.710812 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:18.710774    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6" podUID="05e376de-c2b1-4b26-9d67-2b9fa6be5752" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:17:20.587514 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:20.587486    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"]
Sep 16 19:17:20.587924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:20.587709    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn" podUID="eb3d6380-1448-4622-aabc-627adde20eb8" containerName="kserve-container" containerID="cri-o://89cd54984a31b302b1c077fc6f8ff15c9085ebdb63539cb40f9ef95228330337" gracePeriod=30
Sep 16 19:17:20.587924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:20.587773    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn" podUID="eb3d6380-1448-4622-aabc-627adde20eb8" containerName="kube-rbac-proxy" containerID="cri-o://4f23ff785de6fcfddfd104c58206302623e852d250abd92870e13c5585217efd" gracePeriod=30
Sep 16 19:17:20.926463 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:20.926385    2554 generic.go:356] "Generic (PLEG): container finished" podID="eb3d6380-1448-4622-aabc-627adde20eb8" containerID="4f23ff785de6fcfddfd104c58206302623e852d250abd92870e13c5585217efd" exitCode=2
Sep 16 19:17:20.926596 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:20.926459    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn" event={"ID":"eb3d6380-1448-4622-aabc-627adde20eb8","Type":"ContainerDied","Data":"4f23ff785de6fcfddfd104c58206302623e852d250abd92870e13c5585217efd"}
Sep 16 19:17:23.518070 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:23.518033    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn" podUID="eb3d6380-1448-4622-aabc-627adde20eb8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Sep 16 19:17:23.523371 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:23.523345    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn" podUID="eb3d6380-1448-4622-aabc-627adde20eb8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Sep 16 19:17:24.915992 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:24.915968    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"
Sep 16 19:17:24.949000 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:24.948974    2554 generic.go:356] "Generic (PLEG): container finished" podID="eb3d6380-1448-4622-aabc-627adde20eb8" containerID="89cd54984a31b302b1c077fc6f8ff15c9085ebdb63539cb40f9ef95228330337" exitCode=0
Sep 16 19:17:24.949130 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:24.949022    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"
Sep 16 19:17:24.949130 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:24.949055    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn" event={"ID":"eb3d6380-1448-4622-aabc-627adde20eb8","Type":"ContainerDied","Data":"89cd54984a31b302b1c077fc6f8ff15c9085ebdb63539cb40f9ef95228330337"}
Sep 16 19:17:24.949130 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:24.949120    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn" event={"ID":"eb3d6380-1448-4622-aabc-627adde20eb8","Type":"ContainerDied","Data":"7e552a9d3eb29a29785eb1151e63bc25cfc991a4f63fb859ee6a2b4792c8da07"}
Sep 16 19:17:24.949305 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:24.949159    2554 scope.go:117] "RemoveContainer" containerID="4f23ff785de6fcfddfd104c58206302623e852d250abd92870e13c5585217efd"
Sep 16 19:17:24.950179 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:24.950156    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb3d6380-1448-4622-aabc-627adde20eb8-cabundle-cert\") pod \"eb3d6380-1448-4622-aabc-627adde20eb8\" (UID: \"eb3d6380-1448-4622-aabc-627adde20eb8\") "
Sep 16 19:17:24.950234 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:24.950193    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb3d6380-1448-4622-aabc-627adde20eb8-kube-rbac-proxy-sar-config\") pod \"eb3d6380-1448-4622-aabc-627adde20eb8\" (UID: \"eb3d6380-1448-4622-aabc-627adde20eb8\") "
Sep 16 19:17:24.950234 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:24.950228    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb3d6380-1448-4622-aabc-627adde20eb8-kserve-provision-location\") pod \"eb3d6380-1448-4622-aabc-627adde20eb8\" (UID: \"eb3d6380-1448-4622-aabc-627adde20eb8\") "
Sep 16 19:17:24.950575 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:24.950389    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdlh8\" (UniqueName: \"kubernetes.io/projected/eb3d6380-1448-4622-aabc-627adde20eb8-kube-api-access-mdlh8\") pod \"eb3d6380-1448-4622-aabc-627adde20eb8\" (UID: \"eb3d6380-1448-4622-aabc-627adde20eb8\") "
Sep 16 19:17:24.950575 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:24.950503    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb3d6380-1448-4622-aabc-627adde20eb8-proxy-tls\") pod \"eb3d6380-1448-4622-aabc-627adde20eb8\" (UID: \"eb3d6380-1448-4622-aabc-627adde20eb8\") "
Sep 16 19:17:24.950575 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:24.950530    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eb3d6380-1448-4622-aabc-627adde20eb8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eb3d6380-1448-4622-aabc-627adde20eb8" (UID: "eb3d6380-1448-4622-aabc-627adde20eb8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:24.950806 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:24.950572    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb3d6380-1448-4622-aabc-627adde20eb8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eb3d6380-1448-4622-aabc-627adde20eb8" (UID: "eb3d6380-1448-4622-aabc-627adde20eb8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:24.950806 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:24.950620    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb3d6380-1448-4622-aabc-627adde20eb8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eb3d6380-1448-4622-aabc-627adde20eb8" (UID: "eb3d6380-1448-4622-aabc-627adde20eb8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:24.950924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:24.950851    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eb3d6380-1448-4622-aabc-627adde20eb8-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:17:24.950924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:24.950871    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eb3d6380-1448-4622-aabc-627adde20eb8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:17:24.950924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:24.950884    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eb3d6380-1448-4622-aabc-627adde20eb8-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:17:24.952706 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:24.952676    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb3d6380-1448-4622-aabc-627adde20eb8-kube-api-access-mdlh8" (OuterVolumeSpecName: "kube-api-access-mdlh8") pod "eb3d6380-1448-4622-aabc-627adde20eb8" (UID: "eb3d6380-1448-4622-aabc-627adde20eb8"). InnerVolumeSpecName "kube-api-access-mdlh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:17:24.953175 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:24.953143    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb3d6380-1448-4622-aabc-627adde20eb8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eb3d6380-1448-4622-aabc-627adde20eb8" (UID: "eb3d6380-1448-4622-aabc-627adde20eb8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:17:24.967433 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:24.967417    2554 scope.go:117] "RemoveContainer" containerID="89cd54984a31b302b1c077fc6f8ff15c9085ebdb63539cb40f9ef95228330337"
Sep 16 19:17:24.974731 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:24.974700    2554 scope.go:117] "RemoveContainer" containerID="3a590ff3efd754277f284699eb3daa7f3b1b08c238544df49c7eca86126dbdf8"
Sep 16 19:17:24.983081 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:24.983062    2554 scope.go:117] "RemoveContainer" containerID="4f23ff785de6fcfddfd104c58206302623e852d250abd92870e13c5585217efd"
Sep 16 19:17:24.983356 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:17:24.983339    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f23ff785de6fcfddfd104c58206302623e852d250abd92870e13c5585217efd\": container with ID starting with 4f23ff785de6fcfddfd104c58206302623e852d250abd92870e13c5585217efd not found: ID does not exist" containerID="4f23ff785de6fcfddfd104c58206302623e852d250abd92870e13c5585217efd"
Sep 16 19:17:24.983403 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:24.983363    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f23ff785de6fcfddfd104c58206302623e852d250abd92870e13c5585217efd"} err="failed to get container status \"4f23ff785de6fcfddfd104c58206302623e852d250abd92870e13c5585217efd\": rpc error: code = NotFound desc = could not find container \"4f23ff785de6fcfddfd104c58206302623e852d250abd92870e13c5585217efd\": container with ID starting with 4f23ff785de6fcfddfd104c58206302623e852d250abd92870e13c5585217efd not found: ID does not exist"
Sep 16 19:17:24.983403 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:24.983379    2554 scope.go:117] "RemoveContainer" containerID="89cd54984a31b302b1c077fc6f8ff15c9085ebdb63539cb40f9ef95228330337"
Sep 16 19:17:24.983617 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:17:24.983600    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89cd54984a31b302b1c077fc6f8ff15c9085ebdb63539cb40f9ef95228330337\": container with ID starting with 89cd54984a31b302b1c077fc6f8ff15c9085ebdb63539cb40f9ef95228330337 not found: ID does not exist" containerID="89cd54984a31b302b1c077fc6f8ff15c9085ebdb63539cb40f9ef95228330337"
Sep 16 19:17:24.983659 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:24.983622    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89cd54984a31b302b1c077fc6f8ff15c9085ebdb63539cb40f9ef95228330337"} err="failed to get container status \"89cd54984a31b302b1c077fc6f8ff15c9085ebdb63539cb40f9ef95228330337\": rpc error: code = NotFound desc = could not find container \"89cd54984a31b302b1c077fc6f8ff15c9085ebdb63539cb40f9ef95228330337\": container with ID starting with 89cd54984a31b302b1c077fc6f8ff15c9085ebdb63539cb40f9ef95228330337 not found: ID does not exist"
Sep 16 19:17:24.983659 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:24.983636    2554 scope.go:117] "RemoveContainer" containerID="3a590ff3efd754277f284699eb3daa7f3b1b08c238544df49c7eca86126dbdf8"
Sep 16 19:17:24.983893 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:17:24.983875    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a590ff3efd754277f284699eb3daa7f3b1b08c238544df49c7eca86126dbdf8\": container with ID starting with 3a590ff3efd754277f284699eb3daa7f3b1b08c238544df49c7eca86126dbdf8 not found: ID does not exist" containerID="3a590ff3efd754277f284699eb3daa7f3b1b08c238544df49c7eca86126dbdf8"
Sep 16 19:17:24.983952 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:24.983899    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a590ff3efd754277f284699eb3daa7f3b1b08c238544df49c7eca86126dbdf8"} err="failed to get container status \"3a590ff3efd754277f284699eb3daa7f3b1b08c238544df49c7eca86126dbdf8\": rpc error: code = NotFound desc = could not find container \"3a590ff3efd754277f284699eb3daa7f3b1b08c238544df49c7eca86126dbdf8\": container with ID starting with 3a590ff3efd754277f284699eb3daa7f3b1b08c238544df49c7eca86126dbdf8 not found: ID does not exist"
Sep 16 19:17:25.051454 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:25.051403    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdlh8\" (UniqueName: \"kubernetes.io/projected/eb3d6380-1448-4622-aabc-627adde20eb8-kube-api-access-mdlh8\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:17:25.051454 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:25.051426    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eb3d6380-1448-4622-aabc-627adde20eb8-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:17:25.270833 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:25.270807    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"]
Sep 16 19:17:25.275756 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:25.275732    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-9ffcf5487-l68bn"]
Sep 16 19:17:26.443546 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:26.443515    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb3d6380-1448-4622-aabc-627adde20eb8" path="/var/lib/kubelet/pods/eb3d6380-1448-4622-aabc-627adde20eb8/volumes"
Sep 16 19:17:28.710026 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:28.709993    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6" podUID="05e376de-c2b1-4b26-9d67-2b9fa6be5752" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:17:38.710963 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:38.710923    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6" podUID="05e376de-c2b1-4b26-9d67-2b9fa6be5752" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Sep 16 19:17:46.956126 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:46.956037    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"]
Sep 16 19:17:46.956556 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:46.956352    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6" podUID="05e376de-c2b1-4b26-9d67-2b9fa6be5752" containerName="kserve-container" containerID="cri-o://552c4c1da507db2676f264753c12e069514faee0b32f9f798edfdeca12584f5f" gracePeriod=30
Sep 16 19:17:46.956556 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:46.956404    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6" podUID="05e376de-c2b1-4b26-9d67-2b9fa6be5752" containerName="kube-rbac-proxy" containerID="cri-o://f5c56ed7b32fbbc2c673eb0acc59d45a6a7e5f866568a8b04cbe8239c29d1eb4" gracePeriod=30
Sep 16 19:17:46.957122 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:46.956844    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"
Sep 16 19:17:48.066931 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:48.066900    2554 generic.go:356] "Generic (PLEG): container finished" podID="05e376de-c2b1-4b26-9d67-2b9fa6be5752" containerID="f5c56ed7b32fbbc2c673eb0acc59d45a6a7e5f866568a8b04cbe8239c29d1eb4" exitCode=2
Sep 16 19:17:48.067318 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:48.066960    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6" event={"ID":"05e376de-c2b1-4b26-9d67-2b9fa6be5752","Type":"ContainerDied","Data":"f5c56ed7b32fbbc2c673eb0acc59d45a6a7e5f866568a8b04cbe8239c29d1eb4"}
Sep 16 19:17:48.705320 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:48.705285    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6" podUID="05e376de-c2b1-4b26-9d67-2b9fa6be5752" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Sep 16 19:17:50.900766 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:50.900746    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"
Sep 16 19:17:50.950647 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:50.950622    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqhfb\" (UniqueName: \"kubernetes.io/projected/05e376de-c2b1-4b26-9d67-2b9fa6be5752-kube-api-access-sqhfb\") pod \"05e376de-c2b1-4b26-9d67-2b9fa6be5752\" (UID: \"05e376de-c2b1-4b26-9d67-2b9fa6be5752\") "
Sep 16 19:17:50.950786 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:50.950739    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05e376de-c2b1-4b26-9d67-2b9fa6be5752-kserve-provision-location\") pod \"05e376de-c2b1-4b26-9d67-2b9fa6be5752\" (UID: \"05e376de-c2b1-4b26-9d67-2b9fa6be5752\") "
Sep 16 19:17:50.950786 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:50.950768    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05e376de-c2b1-4b26-9d67-2b9fa6be5752-proxy-tls\") pod \"05e376de-c2b1-4b26-9d67-2b9fa6be5752\" (UID: \"05e376de-c2b1-4b26-9d67-2b9fa6be5752\") "
Sep 16 19:17:50.950906 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:50.950808    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05e376de-c2b1-4b26-9d67-2b9fa6be5752-kube-rbac-proxy-sar-config\") pod \"05e376de-c2b1-4b26-9d67-2b9fa6be5752\" (UID: \"05e376de-c2b1-4b26-9d67-2b9fa6be5752\") "
Sep 16 19:17:50.950906 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:50.950827    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05e376de-c2b1-4b26-9d67-2b9fa6be5752-cabundle-cert\") pod \"05e376de-c2b1-4b26-9d67-2b9fa6be5752\" (UID: \"05e376de-c2b1-4b26-9d67-2b9fa6be5752\") "
Sep 16 19:17:50.951107 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:50.951074    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05e376de-c2b1-4b26-9d67-2b9fa6be5752-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "05e376de-c2b1-4b26-9d67-2b9fa6be5752" (UID: "05e376de-c2b1-4b26-9d67-2b9fa6be5752"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:17:50.951199 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:50.951150    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05e376de-c2b1-4b26-9d67-2b9fa6be5752-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "05e376de-c2b1-4b26-9d67-2b9fa6be5752" (UID: "05e376de-c2b1-4b26-9d67-2b9fa6be5752"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:50.951199 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:50.951157    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05e376de-c2b1-4b26-9d67-2b9fa6be5752-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "05e376de-c2b1-4b26-9d67-2b9fa6be5752" (UID: "05e376de-c2b1-4b26-9d67-2b9fa6be5752"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:17:50.952614 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:50.952592    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05e376de-c2b1-4b26-9d67-2b9fa6be5752-kube-api-access-sqhfb" (OuterVolumeSpecName: "kube-api-access-sqhfb") pod "05e376de-c2b1-4b26-9d67-2b9fa6be5752" (UID: "05e376de-c2b1-4b26-9d67-2b9fa6be5752"). InnerVolumeSpecName "kube-api-access-sqhfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:17:50.952704 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:50.952633    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05e376de-c2b1-4b26-9d67-2b9fa6be5752-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "05e376de-c2b1-4b26-9d67-2b9fa6be5752" (UID: "05e376de-c2b1-4b26-9d67-2b9fa6be5752"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:17:51.052037 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:51.051982    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/05e376de-c2b1-4b26-9d67-2b9fa6be5752-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.052037 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:51.052004    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05e376de-c2b1-4b26-9d67-2b9fa6be5752-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.052037 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:51.052013    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/05e376de-c2b1-4b26-9d67-2b9fa6be5752-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.052037 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:51.052022    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/05e376de-c2b1-4b26-9d67-2b9fa6be5752-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.052037 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:51.052030    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqhfb\" (UniqueName: \"kubernetes.io/projected/05e376de-c2b1-4b26-9d67-2b9fa6be5752-kube-api-access-sqhfb\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:17:51.083302 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:51.083280    2554 generic.go:356] "Generic (PLEG): container finished" podID="05e376de-c2b1-4b26-9d67-2b9fa6be5752" containerID="552c4c1da507db2676f264753c12e069514faee0b32f9f798edfdeca12584f5f" exitCode=0
Sep 16 19:17:51.083403 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:51.083339    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"
Sep 16 19:17:51.083445 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:51.083339    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6" event={"ID":"05e376de-c2b1-4b26-9d67-2b9fa6be5752","Type":"ContainerDied","Data":"552c4c1da507db2676f264753c12e069514faee0b32f9f798edfdeca12584f5f"}
Sep 16 19:17:51.083499 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:51.083479    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6" event={"ID":"05e376de-c2b1-4b26-9d67-2b9fa6be5752","Type":"ContainerDied","Data":"1bb1ea4ea2dac0d8deabd6a8f301604288d108285d065f6451e9990f90ae1c10"}
Sep 16 19:17:51.083533 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:51.083519    2554 scope.go:117] "RemoveContainer" containerID="f5c56ed7b32fbbc2c673eb0acc59d45a6a7e5f866568a8b04cbe8239c29d1eb4"
Sep 16 19:17:51.092158 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:51.092140    2554 scope.go:117] "RemoveContainer" containerID="552c4c1da507db2676f264753c12e069514faee0b32f9f798edfdeca12584f5f"
Sep 16 19:17:51.099567 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:51.099549    2554 scope.go:117] "RemoveContainer" containerID="ce581ff56e24cc959976b36410f4767a1f5c943b7012c9041cb0fab9ecdeac6f"
Sep 16 19:17:51.107529 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:51.107501    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"]
Sep 16 19:17:51.110624 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:51.108997    2554 scope.go:117] "RemoveContainer" containerID="f5c56ed7b32fbbc2c673eb0acc59d45a6a7e5f866568a8b04cbe8239c29d1eb4"
Sep 16 19:17:51.110755 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:17:51.110665    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5c56ed7b32fbbc2c673eb0acc59d45a6a7e5f866568a8b04cbe8239c29d1eb4\": container with ID starting with f5c56ed7b32fbbc2c673eb0acc59d45a6a7e5f866568a8b04cbe8239c29d1eb4 not found: ID does not exist" containerID="f5c56ed7b32fbbc2c673eb0acc59d45a6a7e5f866568a8b04cbe8239c29d1eb4"
Sep 16 19:17:51.110755 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:51.110696    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c56ed7b32fbbc2c673eb0acc59d45a6a7e5f866568a8b04cbe8239c29d1eb4"} err="failed to get container status \"f5c56ed7b32fbbc2c673eb0acc59d45a6a7e5f866568a8b04cbe8239c29d1eb4\": rpc error: code = NotFound desc = could not find container \"f5c56ed7b32fbbc2c673eb0acc59d45a6a7e5f866568a8b04cbe8239c29d1eb4\": container with ID starting with f5c56ed7b32fbbc2c673eb0acc59d45a6a7e5f866568a8b04cbe8239c29d1eb4 not found: ID does not exist"
Sep 16 19:17:51.110755 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:51.110738    2554 scope.go:117] "RemoveContainer" containerID="552c4c1da507db2676f264753c12e069514faee0b32f9f798edfdeca12584f5f"
Sep 16 19:17:51.111407 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:17:51.111377    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"552c4c1da507db2676f264753c12e069514faee0b32f9f798edfdeca12584f5f\": container with ID starting with 552c4c1da507db2676f264753c12e069514faee0b32f9f798edfdeca12584f5f not found: ID does not exist" containerID="552c4c1da507db2676f264753c12e069514faee0b32f9f798edfdeca12584f5f"
Sep 16 19:17:51.111493 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:51.111411    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"552c4c1da507db2676f264753c12e069514faee0b32f9f798edfdeca12584f5f"} err="failed to get container status \"552c4c1da507db2676f264753c12e069514faee0b32f9f798edfdeca12584f5f\": rpc error: code = NotFound desc = could not find container \"552c4c1da507db2676f264753c12e069514faee0b32f9f798edfdeca12584f5f\": container with ID starting with 552c4c1da507db2676f264753c12e069514faee0b32f9f798edfdeca12584f5f not found: ID does not exist"
Sep 16 19:17:51.111493 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:51.111432    2554 scope.go:117] "RemoveContainer" containerID="ce581ff56e24cc959976b36410f4767a1f5c943b7012c9041cb0fab9ecdeac6f"
Sep 16 19:17:51.111797 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:17:51.111779    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce581ff56e24cc959976b36410f4767a1f5c943b7012c9041cb0fab9ecdeac6f\": container with ID starting with ce581ff56e24cc959976b36410f4767a1f5c943b7012c9041cb0fab9ecdeac6f not found: ID does not exist" containerID="ce581ff56e24cc959976b36410f4767a1f5c943b7012c9041cb0fab9ecdeac6f"
Sep 16 19:17:51.111878 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:51.111807    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce581ff56e24cc959976b36410f4767a1f5c943b7012c9041cb0fab9ecdeac6f"} err="failed to get container status \"ce581ff56e24cc959976b36410f4767a1f5c943b7012c9041cb0fab9ecdeac6f\": rpc error: code = NotFound desc = could not find container \"ce581ff56e24cc959976b36410f4767a1f5c943b7012c9041cb0fab9ecdeac6f\": container with ID starting with ce581ff56e24cc959976b36410f4767a1f5c943b7012c9041cb0fab9ecdeac6f not found: ID does not exist"
Sep 16 19:17:51.111958 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:51.111942    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-649dd98487-bfvz6"]
Sep 16 19:17:52.443122 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:52.443078    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05e376de-c2b1-4b26-9d67-2b9fa6be5752" path="/var/lib/kubelet/pods/05e376de-c2b1-4b26-9d67-2b9fa6be5752/volumes"
Sep 16 19:17:55.020048 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.020012    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5"]
Sep 16 19:17:55.020520 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.020500    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb3d6380-1448-4622-aabc-627adde20eb8" containerName="storage-initializer"
Sep 16 19:17:55.020598 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.020522    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb3d6380-1448-4622-aabc-627adde20eb8" containerName="storage-initializer"
Sep 16 19:17:55.020598 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.020534    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05e376de-c2b1-4b26-9d67-2b9fa6be5752" containerName="storage-initializer"
Sep 16 19:17:55.020598 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.020542    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="05e376de-c2b1-4b26-9d67-2b9fa6be5752" containerName="storage-initializer"
Sep 16 19:17:55.020598 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.020567    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05e376de-c2b1-4b26-9d67-2b9fa6be5752" containerName="kube-rbac-proxy"
Sep 16 19:17:55.020598 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.020576    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="05e376de-c2b1-4b26-9d67-2b9fa6be5752" containerName="kube-rbac-proxy"
Sep 16 19:17:55.020598 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.020595    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05e376de-c2b1-4b26-9d67-2b9fa6be5752" containerName="kserve-container"
Sep 16 19:17:55.020945 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.020605    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="05e376de-c2b1-4b26-9d67-2b9fa6be5752" containerName="kserve-container"
Sep 16 19:17:55.020945 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.020630    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb3d6380-1448-4622-aabc-627adde20eb8" containerName="kserve-container"
Sep 16 19:17:55.020945 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.020639    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb3d6380-1448-4622-aabc-627adde20eb8" containerName="kserve-container"
Sep 16 19:17:55.020945 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.020650    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb3d6380-1448-4622-aabc-627adde20eb8" containerName="kube-rbac-proxy"
Sep 16 19:17:55.020945 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.020659    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb3d6380-1448-4622-aabc-627adde20eb8" containerName="kube-rbac-proxy"
Sep 16 19:17:55.020945 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.020765    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb3d6380-1448-4622-aabc-627adde20eb8" containerName="kube-rbac-proxy"
Sep 16 19:17:55.020945 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.020779    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb3d6380-1448-4622-aabc-627adde20eb8" containerName="kserve-container"
Sep 16 19:17:55.020945 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.020788    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="05e376de-c2b1-4b26-9d67-2b9fa6be5752" containerName="kserve-container"
Sep 16 19:17:55.020945 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.020799    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="05e376de-c2b1-4b26-9d67-2b9fa6be5752" containerName="kube-rbac-proxy"
Sep 16 19:17:55.024350 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.024330    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5"
Sep 16 19:17:55.028235 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.028216    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:17:55.028777 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.028754    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:17:55.028892 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.028760    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:17:55.028991 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.028978    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8gklb\""
Sep 16 19:17:55.029047 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.029035    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 16 19:17:55.029137 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.029122    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 16 19:17:55.032654 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.032631    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:17:55.046611 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.046592    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5"]
Sep 16 19:17:55.088015 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.087992    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d49bf999-9dc9-4593-bd57-a66689e15cae-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5\" (UID: \"d49bf999-9dc9-4593-bd57-a66689e15cae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5"
Sep 16 19:17:55.088117 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.088022    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d49bf999-9dc9-4593-bd57-a66689e15cae-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5\" (UID: \"d49bf999-9dc9-4593-bd57-a66689e15cae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5"
Sep 16 19:17:55.088117 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.088095    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d49bf999-9dc9-4593-bd57-a66689e15cae-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5\" (UID: \"d49bf999-9dc9-4593-bd57-a66689e15cae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5"
Sep 16 19:17:55.088243 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.088132    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pfbs\" (UniqueName: \"kubernetes.io/projected/d49bf999-9dc9-4593-bd57-a66689e15cae-kube-api-access-7pfbs\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5\" (UID: \"d49bf999-9dc9-4593-bd57-a66689e15cae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5"
Sep 16 19:17:55.088243 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.088198    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d49bf999-9dc9-4593-bd57-a66689e15cae-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5\" (UID: \"d49bf999-9dc9-4593-bd57-a66689e15cae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5"
Sep 16 19:17:55.189506 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.189474    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d49bf999-9dc9-4593-bd57-a66689e15cae-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5\" (UID: \"d49bf999-9dc9-4593-bd57-a66689e15cae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5"
Sep 16 19:17:55.189604 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.189512    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pfbs\" (UniqueName: \"kubernetes.io/projected/d49bf999-9dc9-4593-bd57-a66689e15cae-kube-api-access-7pfbs\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5\" (UID: \"d49bf999-9dc9-4593-bd57-a66689e15cae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5"
Sep 16 19:17:55.189604 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.189581    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d49bf999-9dc9-4593-bd57-a66689e15cae-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5\" (UID: \"d49bf999-9dc9-4593-bd57-a66689e15cae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5"
Sep 16 19:17:55.189771 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.189622    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d49bf999-9dc9-4593-bd57-a66689e15cae-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5\" (UID: \"d49bf999-9dc9-4593-bd57-a66689e15cae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5"
Sep 16 19:17:55.189771 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.189650    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d49bf999-9dc9-4593-bd57-a66689e15cae-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5\" (UID: \"d49bf999-9dc9-4593-bd57-a66689e15cae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5"
Sep 16 19:17:55.190004 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.189977    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d49bf999-9dc9-4593-bd57-a66689e15cae-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5\" (UID: \"d49bf999-9dc9-4593-bd57-a66689e15cae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5"
Sep 16 19:17:55.190248 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.190226    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d49bf999-9dc9-4593-bd57-a66689e15cae-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5\" (UID: \"d49bf999-9dc9-4593-bd57-a66689e15cae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5"
Sep 16 19:17:55.190325 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.190255    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d49bf999-9dc9-4593-bd57-a66689e15cae-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5\" (UID: \"d49bf999-9dc9-4593-bd57-a66689e15cae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5"
Sep 16 19:17:55.191906 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.191887    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d49bf999-9dc9-4593-bd57-a66689e15cae-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5\" (UID: \"d49bf999-9dc9-4593-bd57-a66689e15cae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5"
Sep 16 19:17:55.198579 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.198558    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pfbs\" (UniqueName: \"kubernetes.io/projected/d49bf999-9dc9-4593-bd57-a66689e15cae-kube-api-access-7pfbs\") pod \"isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5\" (UID: \"d49bf999-9dc9-4593-bd57-a66689e15cae\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5"
Sep 16 19:17:55.334682 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.334628    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5"
Sep 16 19:17:55.627669 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:55.627637    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5"]
Sep 16 19:17:55.629268 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:17:55.629244    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49bf999_9dc9_4593_bd57_a66689e15cae.slice/crio-44382c64d1f26bc7d78b1aebb2d8397041579fe348010eb928f593a337f94562 WatchSource:0}: Error finding container 44382c64d1f26bc7d78b1aebb2d8397041579fe348010eb928f593a337f94562: Status 404 returned error can't find the container with id 44382c64d1f26bc7d78b1aebb2d8397041579fe348010eb928f593a337f94562
Sep 16 19:17:56.108053 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:56.108019    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5" event={"ID":"d49bf999-9dc9-4593-bd57-a66689e15cae","Type":"ContainerStarted","Data":"11fef9d2f99d8c2a6960182dca78e647365b3ff5b8a944a70737d4fc919ff57a"}
Sep 16 19:17:56.108496 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:17:56.108076    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5" event={"ID":"d49bf999-9dc9-4593-bd57-a66689e15cae","Type":"ContainerStarted","Data":"44382c64d1f26bc7d78b1aebb2d8397041579fe348010eb928f593a337f94562"}
Sep 16 19:18:00.128711 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:00.128674    2554 generic.go:356] "Generic (PLEG): container finished" podID="d49bf999-9dc9-4593-bd57-a66689e15cae" containerID="11fef9d2f99d8c2a6960182dca78e647365b3ff5b8a944a70737d4fc919ff57a" exitCode=0
Sep 16 19:18:00.129151 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:00.128761    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5" event={"ID":"d49bf999-9dc9-4593-bd57-a66689e15cae","Type":"ContainerDied","Data":"11fef9d2f99d8c2a6960182dca78e647365b3ff5b8a944a70737d4fc919ff57a"}
Sep 16 19:18:01.138852 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:01.138819    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5" event={"ID":"d49bf999-9dc9-4593-bd57-a66689e15cae","Type":"ContainerStarted","Data":"5cd4faad1999518e20daf7e5c39234e28f9a411011fddafc98a2d7a790ea1064"}
Sep 16 19:18:01.139334 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:01.138894    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5" event={"ID":"d49bf999-9dc9-4593-bd57-a66689e15cae","Type":"ContainerStarted","Data":"d92163612b15cc6011192bad5b1dcac9423d32a3f97df055f7a86f8cb2dfd129"}
Sep 16 19:18:01.139334 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:01.139186    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5"
Sep 16 19:18:01.175663 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:01.175616    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5" podStartSLOduration=2.295988345 podStartE2EDuration="6.175600662s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.879612317s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:18:01.175107506 +0000 UTC m=+1991.250877140" watchObservedRunningTime="2026-09-16 19:18:01.175600662 +0000 UTC m=+1991.251370297"
Sep 16 19:18:02.143868 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:02.143835    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5"
Sep 16 19:18:02.144987 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:02.144960    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5" podUID="d49bf999-9dc9-4593-bd57-a66689e15cae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:18:03.148676 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:03.148641    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5" podUID="d49bf999-9dc9-4593-bd57-a66689e15cae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:18:08.154176 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:08.154146    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5"
Sep 16 19:18:08.154623 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:08.154599    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5" podUID="d49bf999-9dc9-4593-bd57-a66689e15cae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:18:11.218772 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:11.218743    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"]
Sep 16 19:18:11.222441 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:11.222424    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"
Sep 16 19:18:11.227260 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:11.227240    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:18:11.227377 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:11.227271    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Sep 16 19:18:11.248672 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:11.248652    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"]
Sep 16 19:18:11.323437 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:11.323415    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daa30e71-6923-463d-aa56-b616d5a85a4e-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-p68m4\" (UID: \"daa30e71-6923-463d-aa56-b616d5a85a4e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"
Sep 16 19:18:11.323539 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:11.323455    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daa30e71-6923-463d-aa56-b616d5a85a4e-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-p68m4\" (UID: \"daa30e71-6923-463d-aa56-b616d5a85a4e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"
Sep 16 19:18:11.323539 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:11.323474    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daa30e71-6923-463d-aa56-b616d5a85a4e-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-p68m4\" (UID: \"daa30e71-6923-463d-aa56-b616d5a85a4e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"
Sep 16 19:18:11.323539 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:11.323522    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daa30e71-6923-463d-aa56-b616d5a85a4e-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-p68m4\" (UID: \"daa30e71-6923-463d-aa56-b616d5a85a4e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"
Sep 16 19:18:11.323655 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:11.323560    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdf5l\" (UniqueName: \"kubernetes.io/projected/daa30e71-6923-463d-aa56-b616d5a85a4e-kube-api-access-vdf5l\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-p68m4\" (UID: \"daa30e71-6923-463d-aa56-b616d5a85a4e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"
Sep 16 19:18:11.424655 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:11.424631    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daa30e71-6923-463d-aa56-b616d5a85a4e-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-p68m4\" (UID: \"daa30e71-6923-463d-aa56-b616d5a85a4e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"
Sep 16 19:18:11.424773 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:11.424659    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daa30e71-6923-463d-aa56-b616d5a85a4e-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-p68m4\" (UID: \"daa30e71-6923-463d-aa56-b616d5a85a4e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"
Sep 16 19:18:11.424773 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:11.424691    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daa30e71-6923-463d-aa56-b616d5a85a4e-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-p68m4\" (UID: \"daa30e71-6923-463d-aa56-b616d5a85a4e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"
Sep 16 19:18:11.424773 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:11.424744    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdf5l\" (UniqueName: \"kubernetes.io/projected/daa30e71-6923-463d-aa56-b616d5a85a4e-kube-api-access-vdf5l\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-p68m4\" (UID: \"daa30e71-6923-463d-aa56-b616d5a85a4e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"
Sep 16 19:18:11.424906 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:11.424777    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daa30e71-6923-463d-aa56-b616d5a85a4e-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-p68m4\" (UID: \"daa30e71-6923-463d-aa56-b616d5a85a4e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"
Sep 16 19:18:11.424906 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:18:11.424788    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-serving-cert: secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Sep 16 19:18:11.424906 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:18:11.424844    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daa30e71-6923-463d-aa56-b616d5a85a4e-proxy-tls podName:daa30e71-6923-463d-aa56-b616d5a85a4e nodeName:}" failed. No retries permitted until 2026-09-16 19:18:11.924823797 +0000 UTC m=+2002.000593414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/daa30e71-6923-463d-aa56-b616d5a85a4e-proxy-tls") pod "isvc-tensorflow-runtime-predictor-855cfcd677-p68m4" (UID: "daa30e71-6923-463d-aa56-b616d5a85a4e") : secret "isvc-tensorflow-runtime-predictor-serving-cert" not found
Sep 16 19:18:11.425093 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:11.425054    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daa30e71-6923-463d-aa56-b616d5a85a4e-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-p68m4\" (UID: \"daa30e71-6923-463d-aa56-b616d5a85a4e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"
Sep 16 19:18:11.425335 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:11.425314    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daa30e71-6923-463d-aa56-b616d5a85a4e-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-p68m4\" (UID: \"daa30e71-6923-463d-aa56-b616d5a85a4e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"
Sep 16 19:18:11.425493 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:11.425474    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daa30e71-6923-463d-aa56-b616d5a85a4e-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-p68m4\" (UID: \"daa30e71-6923-463d-aa56-b616d5a85a4e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"
Sep 16 19:18:11.434442 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:11.434422    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdf5l\" (UniqueName: \"kubernetes.io/projected/daa30e71-6923-463d-aa56-b616d5a85a4e-kube-api-access-vdf5l\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-p68m4\" (UID: \"daa30e71-6923-463d-aa56-b616d5a85a4e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"
Sep 16 19:18:11.927624 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:11.927596    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daa30e71-6923-463d-aa56-b616d5a85a4e-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-p68m4\" (UID: \"daa30e71-6923-463d-aa56-b616d5a85a4e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"
Sep 16 19:18:11.929856 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:11.929831    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daa30e71-6923-463d-aa56-b616d5a85a4e-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-p68m4\" (UID: \"daa30e71-6923-463d-aa56-b616d5a85a4e\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"
Sep 16 19:18:12.132341 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:12.132320    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"
Sep 16 19:18:12.404396 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:12.404363    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"]
Sep 16 19:18:12.405584 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:18:12.405564    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaa30e71_6923_463d_aa56_b616d5a85a4e.slice/crio-b86d2353107cbdde5dff0db77ac133de9065431836f756112257a99aae30045e WatchSource:0}: Error finding container b86d2353107cbdde5dff0db77ac133de9065431836f756112257a99aae30045e: Status 404 returned error can't find the container with id b86d2353107cbdde5dff0db77ac133de9065431836f756112257a99aae30045e
Sep 16 19:18:13.195976 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:13.195944    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4" event={"ID":"daa30e71-6923-463d-aa56-b616d5a85a4e","Type":"ContainerStarted","Data":"b8b4a26d67284277b35ec6540076cd34ce2ed783cbafa023e9b2d217fbbd7777"}
Sep 16 19:18:13.196153 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:13.196000    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4" event={"ID":"daa30e71-6923-463d-aa56-b616d5a85a4e","Type":"ContainerStarted","Data":"b86d2353107cbdde5dff0db77ac133de9065431836f756112257a99aae30045e"}
Sep 16 19:18:18.154863 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:18.154817    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5" podUID="d49bf999-9dc9-4593-bd57-a66689e15cae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:18:18.220225 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:18.220195    2554 generic.go:356] "Generic (PLEG): container finished" podID="daa30e71-6923-463d-aa56-b616d5a85a4e" containerID="b8b4a26d67284277b35ec6540076cd34ce2ed783cbafa023e9b2d217fbbd7777" exitCode=0
Sep 16 19:18:18.220421 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:18.220274    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4" event={"ID":"daa30e71-6923-463d-aa56-b616d5a85a4e","Type":"ContainerDied","Data":"b8b4a26d67284277b35ec6540076cd34ce2ed783cbafa023e9b2d217fbbd7777"}
Sep 16 19:18:18.221520 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:18.221505    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:18:22.244410 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:22.244325    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4" event={"ID":"daa30e71-6923-463d-aa56-b616d5a85a4e","Type":"ContainerStarted","Data":"2f5a113105ab84567e5917c33d0ccdcafde4d68d81514846be2a406530363e95"}
Sep 16 19:18:22.244410 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:22.244385    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4" event={"ID":"daa30e71-6923-463d-aa56-b616d5a85a4e","Type":"ContainerStarted","Data":"acc8c1f1a66a136c1b68afd14433ddeaf3b405959788b528e01e8c92aafc2c73"}
Sep 16 19:18:22.244862 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:22.244577    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"
Sep 16 19:18:22.264254 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:22.264202    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4" podStartSLOduration=2.913531212 podStartE2EDuration="11.264186317s" totalImagesPullingTime="3.580960803s" totalInitContainerRuntime="4.769694302s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:18:22.26228674 +0000 UTC m=+2012.338056371" watchObservedRunningTime="2026-09-16 19:18:22.264186317 +0000 UTC m=+2012.339956002"
Sep 16 19:18:23.249052 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:23.249020    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"
Sep 16 19:18:23.250233 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:23.250199    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4" podUID="daa30e71-6923-463d-aa56-b616d5a85a4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:18:24.254045 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:24.253997    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4" podUID="daa30e71-6923-463d-aa56-b616d5a85a4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:18:28.154912 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:28.154875    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5" podUID="d49bf999-9dc9-4593-bd57-a66689e15cae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:18:29.258786 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:29.258757    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"
Sep 16 19:18:29.259226 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:29.259197    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4" podUID="daa30e71-6923-463d-aa56-b616d5a85a4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Sep 16 19:18:38.154711 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:38.154669    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5" podUID="d49bf999-9dc9-4593-bd57-a66689e15cae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:18:39.260150 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:39.260121    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"
Sep 16 19:18:48.155304 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:48.155263    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5" podUID="d49bf999-9dc9-4593-bd57-a66689e15cae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:18:51.397530 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:51.397496    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"]
Sep 16 19:18:51.397942 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:51.397740    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4" podUID="daa30e71-6923-463d-aa56-b616d5a85a4e" containerName="kserve-container" containerID="cri-o://acc8c1f1a66a136c1b68afd14433ddeaf3b405959788b528e01e8c92aafc2c73" gracePeriod=30
Sep 16 19:18:51.397942 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:51.397785    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4" podUID="daa30e71-6923-463d-aa56-b616d5a85a4e" containerName="kube-rbac-proxy" containerID="cri-o://2f5a113105ab84567e5917c33d0ccdcafde4d68d81514846be2a406530363e95" gracePeriod=30
Sep 16 19:18:52.394685 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:52.394654    2554 generic.go:356] "Generic (PLEG): container finished" podID="daa30e71-6923-463d-aa56-b616d5a85a4e" containerID="2f5a113105ab84567e5917c33d0ccdcafde4d68d81514846be2a406530363e95" exitCode=2
Sep 16 19:18:52.394887 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:52.394743    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4" event={"ID":"daa30e71-6923-463d-aa56-b616d5a85a4e","Type":"ContainerDied","Data":"2f5a113105ab84567e5917c33d0ccdcafde4d68d81514846be2a406530363e95"}
Sep 16 19:18:54.254386 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:54.254351    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4" podUID="daa30e71-6923-463d-aa56-b616d5a85a4e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 16 19:18:58.155570 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:58.155523    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5" podUID="d49bf999-9dc9-4593-bd57-a66689e15cae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:18:59.255054 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:18:59.255015    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4" podUID="daa30e71-6923-463d-aa56-b616d5a85a4e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 16 19:19:04.254783 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:04.254736    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4" podUID="daa30e71-6923-463d-aa56-b616d5a85a4e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 16 19:19:04.255228 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:04.254899    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"
Sep 16 19:19:08.154645 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:08.154594    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5" podUID="d49bf999-9dc9-4593-bd57-a66689e15cae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:19:09.254639 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:09.254601    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4" podUID="daa30e71-6923-463d-aa56-b616d5a85a4e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 16 19:19:14.254234 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:14.254149    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4" podUID="daa30e71-6923-463d-aa56-b616d5a85a4e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 16 19:19:18.155868 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:18.155839    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5"
Sep 16 19:19:19.255127 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:19.255079    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4" podUID="daa30e71-6923-463d-aa56-b616d5a85a4e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Sep 16 19:19:21.531310 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:21.531274    2554 generic.go:356] "Generic (PLEG): container finished" podID="daa30e71-6923-463d-aa56-b616d5a85a4e" containerID="acc8c1f1a66a136c1b68afd14433ddeaf3b405959788b528e01e8c92aafc2c73" exitCode=137
Sep 16 19:19:21.531669 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:21.531329    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4" event={"ID":"daa30e71-6923-463d-aa56-b616d5a85a4e","Type":"ContainerDied","Data":"acc8c1f1a66a136c1b68afd14433ddeaf3b405959788b528e01e8c92aafc2c73"}
Sep 16 19:19:22.041269 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:22.041243    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"
Sep 16 19:19:22.109981 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:22.109936    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daa30e71-6923-463d-aa56-b616d5a85a4e-cabundle-cert\") pod \"daa30e71-6923-463d-aa56-b616d5a85a4e\" (UID: \"daa30e71-6923-463d-aa56-b616d5a85a4e\") "
Sep 16 19:19:22.110182 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:22.110027    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daa30e71-6923-463d-aa56-b616d5a85a4e-kserve-provision-location\") pod \"daa30e71-6923-463d-aa56-b616d5a85a4e\" (UID: \"daa30e71-6923-463d-aa56-b616d5a85a4e\") "
Sep 16 19:19:22.110182 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:22.110157    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daa30e71-6923-463d-aa56-b616d5a85a4e-kube-rbac-proxy-sar-config\") pod \"daa30e71-6923-463d-aa56-b616d5a85a4e\" (UID: \"daa30e71-6923-463d-aa56-b616d5a85a4e\") "
Sep 16 19:19:22.110308 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:22.110193    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdf5l\" (UniqueName: \"kubernetes.io/projected/daa30e71-6923-463d-aa56-b616d5a85a4e-kube-api-access-vdf5l\") pod \"daa30e71-6923-463d-aa56-b616d5a85a4e\" (UID: \"daa30e71-6923-463d-aa56-b616d5a85a4e\") "
Sep 16 19:19:22.110365 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:22.110326    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daa30e71-6923-463d-aa56-b616d5a85a4e-proxy-tls\") pod \"daa30e71-6923-463d-aa56-b616d5a85a4e\" (UID: \"daa30e71-6923-463d-aa56-b616d5a85a4e\") "
Sep 16 19:19:22.110484 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:22.110459    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daa30e71-6923-463d-aa56-b616d5a85a4e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "daa30e71-6923-463d-aa56-b616d5a85a4e" (UID: "daa30e71-6923-463d-aa56-b616d5a85a4e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:22.110589 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:22.110563    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daa30e71-6923-463d-aa56-b616d5a85a4e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "daa30e71-6923-463d-aa56-b616d5a85a4e" (UID: "daa30e71-6923-463d-aa56-b616d5a85a4e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:22.110643 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:22.110627    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/daa30e71-6923-463d-aa56-b616d5a85a4e-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:19:22.112335 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:22.112314    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daa30e71-6923-463d-aa56-b616d5a85a4e-kube-api-access-vdf5l" (OuterVolumeSpecName: "kube-api-access-vdf5l") pod "daa30e71-6923-463d-aa56-b616d5a85a4e" (UID: "daa30e71-6923-463d-aa56-b616d5a85a4e"). InnerVolumeSpecName "kube-api-access-vdf5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:19:22.112456 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:22.112440    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daa30e71-6923-463d-aa56-b616d5a85a4e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "daa30e71-6923-463d-aa56-b616d5a85a4e" (UID: "daa30e71-6923-463d-aa56-b616d5a85a4e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:22.120817 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:22.120793    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/daa30e71-6923-463d-aa56-b616d5a85a4e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "daa30e71-6923-463d-aa56-b616d5a85a4e" (UID: "daa30e71-6923-463d-aa56-b616d5a85a4e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:22.211225 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:22.211138    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdf5l\" (UniqueName: \"kubernetes.io/projected/daa30e71-6923-463d-aa56-b616d5a85a4e-kube-api-access-vdf5l\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:19:22.211225 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:22.211171    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/daa30e71-6923-463d-aa56-b616d5a85a4e-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:19:22.211225 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:22.211183    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/daa30e71-6923-463d-aa56-b616d5a85a4e-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:19:22.211225 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:22.211191    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/daa30e71-6923-463d-aa56-b616d5a85a4e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:19:22.537664 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:22.537628    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4" event={"ID":"daa30e71-6923-463d-aa56-b616d5a85a4e","Type":"ContainerDied","Data":"b86d2353107cbdde5dff0db77ac133de9065431836f756112257a99aae30045e"}
Sep 16 19:19:22.537664 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:22.537663    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"
Sep 16 19:19:22.538136 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:22.537694    2554 scope.go:117] "RemoveContainer" containerID="2f5a113105ab84567e5917c33d0ccdcafde4d68d81514846be2a406530363e95"
Sep 16 19:19:22.545873 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:22.545853    2554 scope.go:117] "RemoveContainer" containerID="acc8c1f1a66a136c1b68afd14433ddeaf3b405959788b528e01e8c92aafc2c73"
Sep 16 19:19:22.553493 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:22.553467    2554 scope.go:117] "RemoveContainer" containerID="b8b4a26d67284277b35ec6540076cd34ce2ed783cbafa023e9b2d217fbbd7777"
Sep 16 19:19:22.555931 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:22.555912    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"]
Sep 16 19:19:22.562582 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:22.562561    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-p68m4"]
Sep 16 19:19:24.443620 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:24.443579    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daa30e71-6923-463d-aa56-b616d5a85a4e" path="/var/lib/kubelet/pods/daa30e71-6923-463d-aa56-b616d5a85a4e/volumes"
Sep 16 19:19:25.159222 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:25.159188    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5"]
Sep 16 19:19:25.159459 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:25.159428    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5" podUID="d49bf999-9dc9-4593-bd57-a66689e15cae" containerName="kserve-container" containerID="cri-o://d92163612b15cc6011192bad5b1dcac9423d32a3f97df055f7a86f8cb2dfd129" gracePeriod=30
Sep 16 19:19:25.159572 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:25.159475    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5" podUID="d49bf999-9dc9-4593-bd57-a66689e15cae" containerName="kube-rbac-proxy" containerID="cri-o://5cd4faad1999518e20daf7e5c39234e28f9a411011fddafc98a2d7a790ea1064" gracePeriod=30
Sep 16 19:19:25.554347 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:25.554319    2554 generic.go:356] "Generic (PLEG): container finished" podID="d49bf999-9dc9-4593-bd57-a66689e15cae" containerID="5cd4faad1999518e20daf7e5c39234e28f9a411011fddafc98a2d7a790ea1064" exitCode=2
Sep 16 19:19:25.554741 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:25.554402    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5" event={"ID":"d49bf999-9dc9-4593-bd57-a66689e15cae","Type":"ContainerDied","Data":"5cd4faad1999518e20daf7e5c39234e28f9a411011fddafc98a2d7a790ea1064"}
Sep 16 19:19:28.149811 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:28.149766    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5" podUID="d49bf999-9dc9-4593-bd57-a66689e15cae" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Sep 16 19:19:28.155373 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:28.155340    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5" podUID="d49bf999-9dc9-4593-bd57-a66689e15cae" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Sep 16 19:19:29.805698 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:29.805675    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5"
Sep 16 19:19:29.879552 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:29.879522    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d49bf999-9dc9-4593-bd57-a66689e15cae-kserve-provision-location\") pod \"d49bf999-9dc9-4593-bd57-a66689e15cae\" (UID: \"d49bf999-9dc9-4593-bd57-a66689e15cae\") "
Sep 16 19:19:29.879687 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:29.879587    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d49bf999-9dc9-4593-bd57-a66689e15cae-cabundle-cert\") pod \"d49bf999-9dc9-4593-bd57-a66689e15cae\" (UID: \"d49bf999-9dc9-4593-bd57-a66689e15cae\") "
Sep 16 19:19:29.879687 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:29.879619    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d49bf999-9dc9-4593-bd57-a66689e15cae-proxy-tls\") pod \"d49bf999-9dc9-4593-bd57-a66689e15cae\" (UID: \"d49bf999-9dc9-4593-bd57-a66689e15cae\") "
Sep 16 19:19:29.879687 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:29.879651    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pfbs\" (UniqueName: \"kubernetes.io/projected/d49bf999-9dc9-4593-bd57-a66689e15cae-kube-api-access-7pfbs\") pod \"d49bf999-9dc9-4593-bd57-a66689e15cae\" (UID: \"d49bf999-9dc9-4593-bd57-a66689e15cae\") "
Sep 16 19:19:29.879687 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:29.879673    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d49bf999-9dc9-4593-bd57-a66689e15cae-kube-rbac-proxy-sar-config\") pod \"d49bf999-9dc9-4593-bd57-a66689e15cae\" (UID: \"d49bf999-9dc9-4593-bd57-a66689e15cae\") "
Sep 16 19:19:29.879938 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:29.879879    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d49bf999-9dc9-4593-bd57-a66689e15cae-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d49bf999-9dc9-4593-bd57-a66689e15cae" (UID: "d49bf999-9dc9-4593-bd57-a66689e15cae"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:19:29.879998 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:29.879953    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d49bf999-9dc9-4593-bd57-a66689e15cae-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d49bf999-9dc9-4593-bd57-a66689e15cae" (UID: "d49bf999-9dc9-4593-bd57-a66689e15cae"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:29.880134 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:29.880113    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d49bf999-9dc9-4593-bd57-a66689e15cae-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d49bf999-9dc9-4593-bd57-a66689e15cae" (UID: "d49bf999-9dc9-4593-bd57-a66689e15cae"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:19:29.881745 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:29.881712    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d49bf999-9dc9-4593-bd57-a66689e15cae-kube-api-access-7pfbs" (OuterVolumeSpecName: "kube-api-access-7pfbs") pod "d49bf999-9dc9-4593-bd57-a66689e15cae" (UID: "d49bf999-9dc9-4593-bd57-a66689e15cae"). InnerVolumeSpecName "kube-api-access-7pfbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:19:29.881864 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:29.881845    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d49bf999-9dc9-4593-bd57-a66689e15cae-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d49bf999-9dc9-4593-bd57-a66689e15cae" (UID: "d49bf999-9dc9-4593-bd57-a66689e15cae"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:19:29.980351 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:29.980288    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d49bf999-9dc9-4593-bd57-a66689e15cae-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.980351 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:29.980311    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d49bf999-9dc9-4593-bd57-a66689e15cae-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.980351 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:29.980325    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pfbs\" (UniqueName: \"kubernetes.io/projected/d49bf999-9dc9-4593-bd57-a66689e15cae-kube-api-access-7pfbs\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.980351 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:29.980337    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d49bf999-9dc9-4593-bd57-a66689e15cae-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:19:29.980351 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:29.980351    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d49bf999-9dc9-4593-bd57-a66689e15cae-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:19:30.454774 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.454749    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw"]
Sep 16 19:19:30.455073 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.455061    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d49bf999-9dc9-4593-bd57-a66689e15cae" containerName="storage-initializer"
Sep 16 19:19:30.455126 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.455075    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d49bf999-9dc9-4593-bd57-a66689e15cae" containerName="storage-initializer"
Sep 16 19:19:30.455126 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.455082    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d49bf999-9dc9-4593-bd57-a66689e15cae" containerName="kube-rbac-proxy"
Sep 16 19:19:30.455126 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.455087    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d49bf999-9dc9-4593-bd57-a66689e15cae" containerName="kube-rbac-proxy"
Sep 16 19:19:30.455126 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.455094    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daa30e71-6923-463d-aa56-b616d5a85a4e" containerName="kserve-container"
Sep 16 19:19:30.455126 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.455099    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="daa30e71-6923-463d-aa56-b616d5a85a4e" containerName="kserve-container"
Sep 16 19:19:30.455126 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.455106    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d49bf999-9dc9-4593-bd57-a66689e15cae" containerName="kserve-container"
Sep 16 19:19:30.455126 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.455110    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d49bf999-9dc9-4593-bd57-a66689e15cae" containerName="kserve-container"
Sep 16 19:19:30.455126 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.455124    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daa30e71-6923-463d-aa56-b616d5a85a4e" containerName="storage-initializer"
Sep 16 19:19:30.455126 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.455129    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="daa30e71-6923-463d-aa56-b616d5a85a4e" containerName="storage-initializer"
Sep 16 19:19:30.455397 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.455136    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daa30e71-6923-463d-aa56-b616d5a85a4e" containerName="kube-rbac-proxy"
Sep 16 19:19:30.455397 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.455141    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="daa30e71-6923-463d-aa56-b616d5a85a4e" containerName="kube-rbac-proxy"
Sep 16 19:19:30.455397 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.455199    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d49bf999-9dc9-4593-bd57-a66689e15cae" containerName="kserve-container"
Sep 16 19:19:30.455397 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.455207    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="daa30e71-6923-463d-aa56-b616d5a85a4e" containerName="kube-rbac-proxy"
Sep 16 19:19:30.455397 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.455215    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d49bf999-9dc9-4593-bd57-a66689e15cae" containerName="kube-rbac-proxy"
Sep 16 19:19:30.455397 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.455220    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="daa30e71-6923-463d-aa56-b616d5a85a4e" containerName="kserve-container"
Sep 16 19:19:30.458380 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.458360    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw"
Sep 16 19:19:30.460111 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.460087    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 16 19:19:30.460360 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.460345    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Sep 16 19:19:30.468142 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.468113    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"]
Sep 16 19:19:30.472085 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.472067    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw"]
Sep 16 19:19:30.472216 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.472195    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"
Sep 16 19:19:30.474588 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.474571    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 16 19:19:30.486628 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.486605    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"]
Sep 16 19:19:30.577969 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.577932    2554 generic.go:356] "Generic (PLEG): container finished" podID="d49bf999-9dc9-4593-bd57-a66689e15cae" containerID="d92163612b15cc6011192bad5b1dcac9423d32a3f97df055f7a86f8cb2dfd129" exitCode=0
Sep 16 19:19:30.578138 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.577997    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5"
Sep 16 19:19:30.578138 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.578013    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5" event={"ID":"d49bf999-9dc9-4593-bd57-a66689e15cae","Type":"ContainerDied","Data":"d92163612b15cc6011192bad5b1dcac9423d32a3f97df055f7a86f8cb2dfd129"}
Sep 16 19:19:30.578138 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.578080    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5" event={"ID":"d49bf999-9dc9-4593-bd57-a66689e15cae","Type":"ContainerDied","Data":"44382c64d1f26bc7d78b1aebb2d8397041579fe348010eb928f593a337f94562"}
Sep 16 19:19:30.578138 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.578107    2554 scope.go:117] "RemoveContainer" containerID="5cd4faad1999518e20daf7e5c39234e28f9a411011fddafc98a2d7a790ea1064"
Sep 16 19:19:30.586654 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.586627    2554 scope.go:117] "RemoveContainer" containerID="d92163612b15cc6011192bad5b1dcac9423d32a3f97df055f7a86f8cb2dfd129"
Sep 16 19:19:30.586756 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.586710    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15187dd7-813a-4e05-8045-3f33dbe02edc-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-lblfw\" (UID: \"15187dd7-813a-4e05-8045-3f33dbe02edc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw"
Sep 16 19:19:30.586804 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.586765    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d2ns\" (UniqueName: \"kubernetes.io/projected/15187dd7-813a-4e05-8045-3f33dbe02edc-kube-api-access-9d2ns\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-lblfw\" (UID: \"15187dd7-813a-4e05-8045-3f33dbe02edc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw"
Sep 16 19:19:30.586804 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.586781    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f686fd5-3880-4fe1-bd55-300065dc46e5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-dtbhn\" (UID: \"2f686fd5-3880-4fe1-bd55-300065dc46e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"
Sep 16 19:19:30.586922 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.586901    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15187dd7-813a-4e05-8045-3f33dbe02edc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-lblfw\" (UID: \"15187dd7-813a-4e05-8045-3f33dbe02edc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw"
Sep 16 19:19:30.586980 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.586937    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15187dd7-813a-4e05-8045-3f33dbe02edc-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-lblfw\" (UID: \"15187dd7-813a-4e05-8045-3f33dbe02edc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw"
Sep 16 19:19:30.587034 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.587018    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f686fd5-3880-4fe1-bd55-300065dc46e5-proxy-tls\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-dtbhn\" (UID: \"2f686fd5-3880-4fe1-bd55-300065dc46e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"
Sep 16 19:19:30.587093 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.587079    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15187dd7-813a-4e05-8045-3f33dbe02edc-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-lblfw\" (UID: \"15187dd7-813a-4e05-8045-3f33dbe02edc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw"
Sep 16 19:19:30.587167 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.587143    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdl8x\" (UniqueName: \"kubernetes.io/projected/2f686fd5-3880-4fe1-bd55-300065dc46e5-kube-api-access-cdl8x\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-dtbhn\" (UID: \"2f686fd5-3880-4fe1-bd55-300065dc46e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"
Sep 16 19:19:30.587207 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.587182    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f686fd5-3880-4fe1-bd55-300065dc46e5-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-dtbhn\" (UID: \"2f686fd5-3880-4fe1-bd55-300065dc46e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"
Sep 16 19:19:30.587253 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.587207    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f686fd5-3880-4fe1-bd55-300065dc46e5-cabundle-cert\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-dtbhn\" (UID: \"2f686fd5-3880-4fe1-bd55-300065dc46e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"
Sep 16 19:19:30.594483 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.594450    2554 scope.go:117] "RemoveContainer" containerID="11fef9d2f99d8c2a6960182dca78e647365b3ff5b8a944a70737d4fc919ff57a"
Sep 16 19:19:30.594622 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.594601    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5"]
Sep 16 19:19:30.599138 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.599119    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-6f55c68fdd-8lwv5"]
Sep 16 19:19:30.602272 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.602257    2554 scope.go:117] "RemoveContainer" containerID="5cd4faad1999518e20daf7e5c39234e28f9a411011fddafc98a2d7a790ea1064"
Sep 16 19:19:30.602516 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:19:30.602500    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cd4faad1999518e20daf7e5c39234e28f9a411011fddafc98a2d7a790ea1064\": container with ID starting with 5cd4faad1999518e20daf7e5c39234e28f9a411011fddafc98a2d7a790ea1064 not found: ID does not exist" containerID="5cd4faad1999518e20daf7e5c39234e28f9a411011fddafc98a2d7a790ea1064"
Sep 16 19:19:30.602557 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.602523    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cd4faad1999518e20daf7e5c39234e28f9a411011fddafc98a2d7a790ea1064"} err="failed to get container status \"5cd4faad1999518e20daf7e5c39234e28f9a411011fddafc98a2d7a790ea1064\": rpc error: code = NotFound desc = could not find container \"5cd4faad1999518e20daf7e5c39234e28f9a411011fddafc98a2d7a790ea1064\": container with ID starting with 5cd4faad1999518e20daf7e5c39234e28f9a411011fddafc98a2d7a790ea1064 not found: ID does not exist"
Sep 16 19:19:30.602557 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.602544    2554 scope.go:117] "RemoveContainer" containerID="d92163612b15cc6011192bad5b1dcac9423d32a3f97df055f7a86f8cb2dfd129"
Sep 16 19:19:30.602828 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:19:30.602810    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d92163612b15cc6011192bad5b1dcac9423d32a3f97df055f7a86f8cb2dfd129\": container with ID starting with d92163612b15cc6011192bad5b1dcac9423d32a3f97df055f7a86f8cb2dfd129 not found: ID does not exist" containerID="d92163612b15cc6011192bad5b1dcac9423d32a3f97df055f7a86f8cb2dfd129"
Sep 16 19:19:30.602889 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.602834    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d92163612b15cc6011192bad5b1dcac9423d32a3f97df055f7a86f8cb2dfd129"} err="failed to get container status \"d92163612b15cc6011192bad5b1dcac9423d32a3f97df055f7a86f8cb2dfd129\": rpc error: code = NotFound desc = could not find container \"d92163612b15cc6011192bad5b1dcac9423d32a3f97df055f7a86f8cb2dfd129\": container with ID starting with d92163612b15cc6011192bad5b1dcac9423d32a3f97df055f7a86f8cb2dfd129 not found: ID does not exist"
Sep 16 19:19:30.602889 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.602847    2554 scope.go:117] "RemoveContainer" containerID="11fef9d2f99d8c2a6960182dca78e647365b3ff5b8a944a70737d4fc919ff57a"
Sep 16 19:19:30.603059 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:19:30.603044    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11fef9d2f99d8c2a6960182dca78e647365b3ff5b8a944a70737d4fc919ff57a\": container with ID starting with 11fef9d2f99d8c2a6960182dca78e647365b3ff5b8a944a70737d4fc919ff57a not found: ID does not exist" containerID="11fef9d2f99d8c2a6960182dca78e647365b3ff5b8a944a70737d4fc919ff57a"
Sep 16 19:19:30.603104 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.603063    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11fef9d2f99d8c2a6960182dca78e647365b3ff5b8a944a70737d4fc919ff57a"} err="failed to get container status \"11fef9d2f99d8c2a6960182dca78e647365b3ff5b8a944a70737d4fc919ff57a\": rpc error: code = NotFound desc = could not find container \"11fef9d2f99d8c2a6960182dca78e647365b3ff5b8a944a70737d4fc919ff57a\": container with ID starting with 11fef9d2f99d8c2a6960182dca78e647365b3ff5b8a944a70737d4fc919ff57a not found: ID does not exist"
Sep 16 19:19:30.687641 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.687595    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15187dd7-813a-4e05-8045-3f33dbe02edc-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-lblfw\" (UID: \"15187dd7-813a-4e05-8045-3f33dbe02edc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw"
Sep 16 19:19:30.687641 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.687650    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdl8x\" (UniqueName: \"kubernetes.io/projected/2f686fd5-3880-4fe1-bd55-300065dc46e5-kube-api-access-cdl8x\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-dtbhn\" (UID: \"2f686fd5-3880-4fe1-bd55-300065dc46e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"
Sep 16 19:19:30.687924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.687666    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f686fd5-3880-4fe1-bd55-300065dc46e5-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-dtbhn\" (UID: \"2f686fd5-3880-4fe1-bd55-300065dc46e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"
Sep 16 19:19:30.687924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.687688    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f686fd5-3880-4fe1-bd55-300065dc46e5-cabundle-cert\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-dtbhn\" (UID: \"2f686fd5-3880-4fe1-bd55-300065dc46e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"
Sep 16 19:19:30.687924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.687843    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15187dd7-813a-4e05-8045-3f33dbe02edc-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-lblfw\" (UID: \"15187dd7-813a-4e05-8045-3f33dbe02edc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw"
Sep 16 19:19:30.687924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.687876    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d2ns\" (UniqueName: \"kubernetes.io/projected/15187dd7-813a-4e05-8045-3f33dbe02edc-kube-api-access-9d2ns\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-lblfw\" (UID: \"15187dd7-813a-4e05-8045-3f33dbe02edc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw"
Sep 16 19:19:30.687924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.687902    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f686fd5-3880-4fe1-bd55-300065dc46e5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-dtbhn\" (UID: \"2f686fd5-3880-4fe1-bd55-300065dc46e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"
Sep 16 19:19:30.688191 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.687997    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15187dd7-813a-4e05-8045-3f33dbe02edc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-lblfw\" (UID: \"15187dd7-813a-4e05-8045-3f33dbe02edc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw"
Sep 16 19:19:30.688191 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.688023    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15187dd7-813a-4e05-8045-3f33dbe02edc-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-lblfw\" (UID: \"15187dd7-813a-4e05-8045-3f33dbe02edc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw"
Sep 16 19:19:30.688191 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.688088    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f686fd5-3880-4fe1-bd55-300065dc46e5-proxy-tls\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-dtbhn\" (UID: \"2f686fd5-3880-4fe1-bd55-300065dc46e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"
Sep 16 19:19:30.688191 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.688154    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f686fd5-3880-4fe1-bd55-300065dc46e5-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-dtbhn\" (UID: \"2f686fd5-3880-4fe1-bd55-300065dc46e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"
Sep 16 19:19:30.688191 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.688163    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15187dd7-813a-4e05-8045-3f33dbe02edc-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-lblfw\" (UID: \"15187dd7-813a-4e05-8045-3f33dbe02edc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw"
Sep 16 19:19:30.688447 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:19:30.688204    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 16 19:19:30.688447 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:19:30.688264    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2f686fd5-3880-4fe1-bd55-300065dc46e5-proxy-tls podName:2f686fd5-3880-4fe1-bd55-300065dc46e5 nodeName:}" failed. No retries permitted until 2026-09-16 19:19:31.188245429 +0000 UTC m=+2081.264015040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2f686fd5-3880-4fe1-bd55-300065dc46e5-proxy-tls") pod "isvc-canary-rollback-predictor-857b7ffb9f-dtbhn" (UID: "2f686fd5-3880-4fe1-bd55-300065dc46e5") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 16 19:19:30.688568 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.688479    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f686fd5-3880-4fe1-bd55-300065dc46e5-cabundle-cert\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-dtbhn\" (UID: \"2f686fd5-3880-4fe1-bd55-300065dc46e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"
Sep 16 19:19:30.688741 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.688706    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15187dd7-813a-4e05-8045-3f33dbe02edc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-lblfw\" (UID: \"15187dd7-813a-4e05-8045-3f33dbe02edc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw"
Sep 16 19:19:30.688822 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.688714    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f686fd5-3880-4fe1-bd55-300065dc46e5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-dtbhn\" (UID: \"2f686fd5-3880-4fe1-bd55-300065dc46e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"
Sep 16 19:19:30.688822 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.688790    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15187dd7-813a-4e05-8045-3f33dbe02edc-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-lblfw\" (UID: \"15187dd7-813a-4e05-8045-3f33dbe02edc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw"
Sep 16 19:19:30.690467 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.690444    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15187dd7-813a-4e05-8045-3f33dbe02edc-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-lblfw\" (UID: \"15187dd7-813a-4e05-8045-3f33dbe02edc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw"
Sep 16 19:19:30.696595 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.696573    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d2ns\" (UniqueName: \"kubernetes.io/projected/15187dd7-813a-4e05-8045-3f33dbe02edc-kube-api-access-9d2ns\") pod \"isvc-canary-rollback-v2-predictor-7794789c9d-lblfw\" (UID: \"15187dd7-813a-4e05-8045-3f33dbe02edc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw"
Sep 16 19:19:30.696901 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.696881    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdl8x\" (UniqueName: \"kubernetes.io/projected/2f686fd5-3880-4fe1-bd55-300065dc46e5-kube-api-access-cdl8x\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-dtbhn\" (UID: \"2f686fd5-3880-4fe1-bd55-300065dc46e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"
Sep 16 19:19:30.768771 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:30.768694    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw"
Sep 16 19:19:31.079215 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:31.079139    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw"]
Sep 16 19:19:31.082338 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:19:31.082313    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15187dd7_813a_4e05_8045_3f33dbe02edc.slice/crio-fc37ee9c00a9f2d7834b08f06b1ed50fe56d9d016a2cf44ad5a8728960db547b WatchSource:0}: Error finding container fc37ee9c00a9f2d7834b08f06b1ed50fe56d9d016a2cf44ad5a8728960db547b: Status 404 returned error can't find the container with id fc37ee9c00a9f2d7834b08f06b1ed50fe56d9d016a2cf44ad5a8728960db547b
Sep 16 19:19:31.192198 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:31.192169    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f686fd5-3880-4fe1-bd55-300065dc46e5-proxy-tls\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-dtbhn\" (UID: \"2f686fd5-3880-4fe1-bd55-300065dc46e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"
Sep 16 19:19:31.194300 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:31.194275    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f686fd5-3880-4fe1-bd55-300065dc46e5-proxy-tls\") pod \"isvc-canary-rollback-predictor-857b7ffb9f-dtbhn\" (UID: \"2f686fd5-3880-4fe1-bd55-300065dc46e5\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"
Sep 16 19:19:31.382106 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:31.382014    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"
Sep 16 19:19:31.584503 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:31.584466    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw" event={"ID":"15187dd7-813a-4e05-8045-3f33dbe02edc","Type":"ContainerStarted","Data":"c8a3c5dac98d14417112f91ae249ddd97b6265d3069177d82b6b7294c319a17c"}
Sep 16 19:19:31.584696 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:31.584550    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw" event={"ID":"15187dd7-813a-4e05-8045-3f33dbe02edc","Type":"ContainerStarted","Data":"fc37ee9c00a9f2d7834b08f06b1ed50fe56d9d016a2cf44ad5a8728960db547b"}
Sep 16 19:19:31.652712 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:31.652688    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"]
Sep 16 19:19:31.655257 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:19:31.655229    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f686fd5_3880_4fe1_bd55_300065dc46e5.slice/crio-c38518d0151aeef578e01da31e356eed2d22711488b21f3a900e6d1f3070f56d WatchSource:0}: Error finding container c38518d0151aeef578e01da31e356eed2d22711488b21f3a900e6d1f3070f56d: Status 404 returned error can't find the container with id c38518d0151aeef578e01da31e356eed2d22711488b21f3a900e6d1f3070f56d
Sep 16 19:19:32.442785 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:32.442744    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d49bf999-9dc9-4593-bd57-a66689e15cae" path="/var/lib/kubelet/pods/d49bf999-9dc9-4593-bd57-a66689e15cae/volumes"
Sep 16 19:19:32.591789 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:32.591753    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn" event={"ID":"2f686fd5-3880-4fe1-bd55-300065dc46e5","Type":"ContainerStarted","Data":"c99b04df88b60cf0f3647490a7c3d2f705fbde00ccda5f955e93578e6b880010"}
Sep 16 19:19:32.591944 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:32.591804    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn" event={"ID":"2f686fd5-3880-4fe1-bd55-300065dc46e5","Type":"ContainerStarted","Data":"c38518d0151aeef578e01da31e356eed2d22711488b21f3a900e6d1f3070f56d"}
Sep 16 19:19:35.609400 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:35.609366    2554 generic.go:356] "Generic (PLEG): container finished" podID="15187dd7-813a-4e05-8045-3f33dbe02edc" containerID="c8a3c5dac98d14417112f91ae249ddd97b6265d3069177d82b6b7294c319a17c" exitCode=0
Sep 16 19:19:35.609797 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:35.609441    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw" event={"ID":"15187dd7-813a-4e05-8045-3f33dbe02edc","Type":"ContainerDied","Data":"c8a3c5dac98d14417112f91ae249ddd97b6265d3069177d82b6b7294c319a17c"}
Sep 16 19:19:36.617363 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:36.617323    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw" event={"ID":"15187dd7-813a-4e05-8045-3f33dbe02edc","Type":"ContainerStarted","Data":"2bec354703b5e9bb49d2334e62142eddf5850cf02d43970bed25cc4c551917db"}
Sep 16 19:19:36.617841 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:36.617395    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw" event={"ID":"15187dd7-813a-4e05-8045-3f33dbe02edc","Type":"ContainerStarted","Data":"996b291efefced85036ab2d7187a3194089592b93decdbcf2757529dd57cd5aa"}
Sep 16 19:19:36.617841 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:36.617598    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw"
Sep 16 19:19:36.618929 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:36.618908    2554 generic.go:356] "Generic (PLEG): container finished" podID="2f686fd5-3880-4fe1-bd55-300065dc46e5" containerID="c99b04df88b60cf0f3647490a7c3d2f705fbde00ccda5f955e93578e6b880010" exitCode=0
Sep 16 19:19:36.619005 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:36.618975    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn" event={"ID":"2f686fd5-3880-4fe1-bd55-300065dc46e5","Type":"ContainerDied","Data":"c99b04df88b60cf0f3647490a7c3d2f705fbde00ccda5f955e93578e6b880010"}
Sep 16 19:19:36.645641 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:36.645596    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw" podStartSLOduration=2.800575449 podStartE2EDuration="6.645583578s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.845008129s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:36.643662923 +0000 UTC m=+2086.719432554" watchObservedRunningTime="2026-09-16 19:19:36.645583578 +0000 UTC m=+2086.721353211"
Sep 16 19:19:37.626082 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:37.626048    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn" event={"ID":"2f686fd5-3880-4fe1-bd55-300065dc46e5","Type":"ContainerStarted","Data":"391d55008cbdcab42f36232f5168b67b5e726377215be1f36baf9fe634679889"}
Sep 16 19:19:37.626509 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:37.626387    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw"
Sep 16 19:19:37.626509 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:37.626427    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn" event={"ID":"2f686fd5-3880-4fe1-bd55-300065dc46e5","Type":"ContainerStarted","Data":"d4a99091bef2e873875e8a5d187aa922ec71ed99bcb1849b340ed389fe1cdcca"}
Sep 16 19:19:37.626639 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:37.626606    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"
Sep 16 19:19:37.627687 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:37.627662    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw" podUID="15187dd7-813a-4e05-8045-3f33dbe02edc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:19:37.651350 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:37.651300    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn" podStartSLOduration=3.591747855 podStartE2EDuration="7.651287446s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.059539591s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:19:37.648922868 +0000 UTC m=+2087.724692513" watchObservedRunningTime="2026-09-16 19:19:37.651287446 +0000 UTC m=+2087.727057078"
Sep 16 19:19:38.631186 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:38.631145    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw" podUID="15187dd7-813a-4e05-8045-3f33dbe02edc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:19:38.631547 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:38.631234    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"
Sep 16 19:19:38.632111 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:38.632086    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn" podUID="2f686fd5-3880-4fe1-bd55-300065dc46e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:19:39.635732 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:39.635674    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn" podUID="2f686fd5-3880-4fe1-bd55-300065dc46e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:19:43.636000 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:43.635968    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw"
Sep 16 19:19:43.642513 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:43.636709    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw" podUID="15187dd7-813a-4e05-8045-3f33dbe02edc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:19:44.641382 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:44.641355    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"
Sep 16 19:19:44.641894 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:44.641863    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn" podUID="2f686fd5-3880-4fe1-bd55-300065dc46e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:19:50.518457 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:50.518430    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/2.log"
Sep 16 19:19:50.523370 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:50.523349    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/2.log"
Sep 16 19:19:50.524910 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:50.524885    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovn-acl-logging/0.log"
Sep 16 19:19:50.529344 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:50.529329    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovn-acl-logging/0.log"
Sep 16 19:19:53.637075 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:53.637034    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw" podUID="15187dd7-813a-4e05-8045-3f33dbe02edc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:19:54.642327 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:19:54.642278    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn" podUID="2f686fd5-3880-4fe1-bd55-300065dc46e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:20:03.636419 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:20:03.636375    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw" podUID="15187dd7-813a-4e05-8045-3f33dbe02edc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:20:04.642439 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:20:04.642399    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn" podUID="2f686fd5-3880-4fe1-bd55-300065dc46e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:20:13.637373 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:20:13.637333    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw" podUID="15187dd7-813a-4e05-8045-3f33dbe02edc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:20:14.642501 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:20:14.642457    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn" podUID="2f686fd5-3880-4fe1-bd55-300065dc46e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:20:23.636600 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:20:23.636560    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw" podUID="15187dd7-813a-4e05-8045-3f33dbe02edc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:20:24.641851 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:20:24.641812    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn" podUID="2f686fd5-3880-4fe1-bd55-300065dc46e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:20:33.636504 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:20:33.636468    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw" podUID="15187dd7-813a-4e05-8045-3f33dbe02edc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:20:34.642466 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:20:34.642427    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn" podUID="2f686fd5-3880-4fe1-bd55-300065dc46e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:20:43.636543 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:20:43.636458    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw" podUID="15187dd7-813a-4e05-8045-3f33dbe02edc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:20:44.642085 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:20:44.642049    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn" podUID="2f686fd5-3880-4fe1-bd55-300065dc46e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:20:52.443666 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:20:52.443627    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"
Sep 16 19:20:53.637101 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:20:53.637073    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw"
Sep 16 19:21:00.451553 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:00.451521    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw"]
Sep 16 19:21:00.452000 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:00.451771    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw" podUID="15187dd7-813a-4e05-8045-3f33dbe02edc" containerName="kserve-container" containerID="cri-o://996b291efefced85036ab2d7187a3194089592b93decdbcf2757529dd57cd5aa" gracePeriod=30
Sep 16 19:21:00.452000 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:00.451785    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw" podUID="15187dd7-813a-4e05-8045-3f33dbe02edc" containerName="kube-rbac-proxy" containerID="cri-o://2bec354703b5e9bb49d2334e62142eddf5850cf02d43970bed25cc4c551917db" gracePeriod=30
Sep 16 19:21:01.027706 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:01.027677    2554 generic.go:356] "Generic (PLEG): container finished" podID="15187dd7-813a-4e05-8045-3f33dbe02edc" containerID="2bec354703b5e9bb49d2334e62142eddf5850cf02d43970bed25cc4c551917db" exitCode=2
Sep 16 19:21:01.027873 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:01.027743    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw" event={"ID":"15187dd7-813a-4e05-8045-3f33dbe02edc","Type":"ContainerDied","Data":"2bec354703b5e9bb49d2334e62142eddf5850cf02d43970bed25cc4c551917db"}
Sep 16 19:21:03.631904 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:03.631862    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw" podUID="15187dd7-813a-4e05-8045-3f33dbe02edc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Sep 16 19:21:03.637244 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:03.637219    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw" podUID="15187dd7-813a-4e05-8045-3f33dbe02edc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Sep 16 19:21:04.495545 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:04.495519    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw"
Sep 16 19:21:04.594669 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:04.594646    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15187dd7-813a-4e05-8045-3f33dbe02edc-cabundle-cert\") pod \"15187dd7-813a-4e05-8045-3f33dbe02edc\" (UID: \"15187dd7-813a-4e05-8045-3f33dbe02edc\") "
Sep 16 19:21:04.594799 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:04.594691    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15187dd7-813a-4e05-8045-3f33dbe02edc-kserve-provision-location\") pod \"15187dd7-813a-4e05-8045-3f33dbe02edc\" (UID: \"15187dd7-813a-4e05-8045-3f33dbe02edc\") "
Sep 16 19:21:04.594799 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:04.594762    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15187dd7-813a-4e05-8045-3f33dbe02edc-proxy-tls\") pod \"15187dd7-813a-4e05-8045-3f33dbe02edc\" (UID: \"15187dd7-813a-4e05-8045-3f33dbe02edc\") "
Sep 16 19:21:04.594799 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:04.594785    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9d2ns\" (UniqueName: \"kubernetes.io/projected/15187dd7-813a-4e05-8045-3f33dbe02edc-kube-api-access-9d2ns\") pod \"15187dd7-813a-4e05-8045-3f33dbe02edc\" (UID: \"15187dd7-813a-4e05-8045-3f33dbe02edc\") "
Sep 16 19:21:04.594922 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:04.594820    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15187dd7-813a-4e05-8045-3f33dbe02edc-kube-rbac-proxy-sar-config\") pod \"15187dd7-813a-4e05-8045-3f33dbe02edc\" (UID: \"15187dd7-813a-4e05-8045-3f33dbe02edc\") "
Sep 16 19:21:04.595020 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:04.594993    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15187dd7-813a-4e05-8045-3f33dbe02edc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "15187dd7-813a-4e05-8045-3f33dbe02edc" (UID: "15187dd7-813a-4e05-8045-3f33dbe02edc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:21:04.595066 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:04.595033    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15187dd7-813a-4e05-8045-3f33dbe02edc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "15187dd7-813a-4e05-8045-3f33dbe02edc" (UID: "15187dd7-813a-4e05-8045-3f33dbe02edc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:04.595251 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:04.595222    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15187dd7-813a-4e05-8045-3f33dbe02edc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "15187dd7-813a-4e05-8045-3f33dbe02edc" (UID: "15187dd7-813a-4e05-8045-3f33dbe02edc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:21:04.596624 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:04.596600    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15187dd7-813a-4e05-8045-3f33dbe02edc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "15187dd7-813a-4e05-8045-3f33dbe02edc" (UID: "15187dd7-813a-4e05-8045-3f33dbe02edc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:21:04.596746 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:04.596703    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15187dd7-813a-4e05-8045-3f33dbe02edc-kube-api-access-9d2ns" (OuterVolumeSpecName: "kube-api-access-9d2ns") pod "15187dd7-813a-4e05-8045-3f33dbe02edc" (UID: "15187dd7-813a-4e05-8045-3f33dbe02edc"). InnerVolumeSpecName "kube-api-access-9d2ns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:21:04.695765 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:04.695743    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15187dd7-813a-4e05-8045-3f33dbe02edc-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:21:04.695765 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:04.695763    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d2ns\" (UniqueName: \"kubernetes.io/projected/15187dd7-813a-4e05-8045-3f33dbe02edc-kube-api-access-9d2ns\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:21:04.696083 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:04.695774    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15187dd7-813a-4e05-8045-3f33dbe02edc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:21:04.696083 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:04.695784    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15187dd7-813a-4e05-8045-3f33dbe02edc-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:21:04.696083 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:04.695793    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15187dd7-813a-4e05-8045-3f33dbe02edc-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:21:05.047863 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:05.047829    2554 generic.go:356] "Generic (PLEG): container finished" podID="15187dd7-813a-4e05-8045-3f33dbe02edc" containerID="996b291efefced85036ab2d7187a3194089592b93decdbcf2757529dd57cd5aa" exitCode=0
Sep 16 19:21:05.047995 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:05.047887    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw"
Sep 16 19:21:05.047995 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:05.047901    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw" event={"ID":"15187dd7-813a-4e05-8045-3f33dbe02edc","Type":"ContainerDied","Data":"996b291efefced85036ab2d7187a3194089592b93decdbcf2757529dd57cd5aa"}
Sep 16 19:21:05.047995 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:05.047961    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw" event={"ID":"15187dd7-813a-4e05-8045-3f33dbe02edc","Type":"ContainerDied","Data":"fc37ee9c00a9f2d7834b08f06b1ed50fe56d9d016a2cf44ad5a8728960db547b"}
Sep 16 19:21:05.047995 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:05.047992    2554 scope.go:117] "RemoveContainer" containerID="2bec354703b5e9bb49d2334e62142eddf5850cf02d43970bed25cc4c551917db"
Sep 16 19:21:05.058136 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:05.058116    2554 scope.go:117] "RemoveContainer" containerID="996b291efefced85036ab2d7187a3194089592b93decdbcf2757529dd57cd5aa"
Sep 16 19:21:05.067186 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:05.067162    2554 scope.go:117] "RemoveContainer" containerID="c8a3c5dac98d14417112f91ae249ddd97b6265d3069177d82b6b7294c319a17c"
Sep 16 19:21:05.075575 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:05.075523    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw"]
Sep 16 19:21:05.077852 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:05.077825    2554 scope.go:117] "RemoveContainer" containerID="2bec354703b5e9bb49d2334e62142eddf5850cf02d43970bed25cc4c551917db"
Sep 16 19:21:05.078148 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:21:05.078120    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bec354703b5e9bb49d2334e62142eddf5850cf02d43970bed25cc4c551917db\": container with ID starting with 2bec354703b5e9bb49d2334e62142eddf5850cf02d43970bed25cc4c551917db not found: ID does not exist" containerID="2bec354703b5e9bb49d2334e62142eddf5850cf02d43970bed25cc4c551917db"
Sep 16 19:21:05.078211 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:05.078159    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bec354703b5e9bb49d2334e62142eddf5850cf02d43970bed25cc4c551917db"} err="failed to get container status \"2bec354703b5e9bb49d2334e62142eddf5850cf02d43970bed25cc4c551917db\": rpc error: code = NotFound desc = could not find container \"2bec354703b5e9bb49d2334e62142eddf5850cf02d43970bed25cc4c551917db\": container with ID starting with 2bec354703b5e9bb49d2334e62142eddf5850cf02d43970bed25cc4c551917db not found: ID does not exist"
Sep 16 19:21:05.078211 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:05.078183    2554 scope.go:117] "RemoveContainer" containerID="996b291efefced85036ab2d7187a3194089592b93decdbcf2757529dd57cd5aa"
Sep 16 19:21:05.078474 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:21:05.078454    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"996b291efefced85036ab2d7187a3194089592b93decdbcf2757529dd57cd5aa\": container with ID starting with 996b291efefced85036ab2d7187a3194089592b93decdbcf2757529dd57cd5aa not found: ID does not exist" containerID="996b291efefced85036ab2d7187a3194089592b93decdbcf2757529dd57cd5aa"
Sep 16 19:21:05.078560 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:05.078483    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"996b291efefced85036ab2d7187a3194089592b93decdbcf2757529dd57cd5aa"} err="failed to get container status \"996b291efefced85036ab2d7187a3194089592b93decdbcf2757529dd57cd5aa\": rpc error: code = NotFound desc = could not find container \"996b291efefced85036ab2d7187a3194089592b93decdbcf2757529dd57cd5aa\": container with ID starting with 996b291efefced85036ab2d7187a3194089592b93decdbcf2757529dd57cd5aa not found: ID does not exist"
Sep 16 19:21:05.078560 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:05.078509    2554 scope.go:117] "RemoveContainer" containerID="c8a3c5dac98d14417112f91ae249ddd97b6265d3069177d82b6b7294c319a17c"
Sep 16 19:21:05.078818 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:21:05.078798    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8a3c5dac98d14417112f91ae249ddd97b6265d3069177d82b6b7294c319a17c\": container with ID starting with c8a3c5dac98d14417112f91ae249ddd97b6265d3069177d82b6b7294c319a17c not found: ID does not exist" containerID="c8a3c5dac98d14417112f91ae249ddd97b6265d3069177d82b6b7294c319a17c"
Sep 16 19:21:05.078893 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:05.078827    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8a3c5dac98d14417112f91ae249ddd97b6265d3069177d82b6b7294c319a17c"} err="failed to get container status \"c8a3c5dac98d14417112f91ae249ddd97b6265d3069177d82b6b7294c319a17c\": rpc error: code = NotFound desc = could not find container \"c8a3c5dac98d14417112f91ae249ddd97b6265d3069177d82b6b7294c319a17c\": container with ID starting with c8a3c5dac98d14417112f91ae249ddd97b6265d3069177d82b6b7294c319a17c not found: ID does not exist"
Sep 16 19:21:05.096103 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:05.096078    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-7794789c9d-lblfw"]
Sep 16 19:21:06.442936 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:06.442907    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15187dd7-813a-4e05-8045-3f33dbe02edc" path="/var/lib/kubelet/pods/15187dd7-813a-4e05-8045-3f33dbe02edc/volumes"
Sep 16 19:21:13.953341 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:13.953304    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"]
Sep 16 19:21:13.953767 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:13.953653    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15187dd7-813a-4e05-8045-3f33dbe02edc" containerName="kserve-container"
Sep 16 19:21:13.953767 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:13.953664    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="15187dd7-813a-4e05-8045-3f33dbe02edc" containerName="kserve-container"
Sep 16 19:21:13.953767 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:13.953681    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15187dd7-813a-4e05-8045-3f33dbe02edc" containerName="kube-rbac-proxy"
Sep 16 19:21:13.953767 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:13.953686    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="15187dd7-813a-4e05-8045-3f33dbe02edc" containerName="kube-rbac-proxy"
Sep 16 19:21:13.953767 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:13.953700    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15187dd7-813a-4e05-8045-3f33dbe02edc" containerName="storage-initializer"
Sep 16 19:21:13.953767 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:13.953705    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="15187dd7-813a-4e05-8045-3f33dbe02edc" containerName="storage-initializer"
Sep 16 19:21:13.953767 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:13.953770    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="15187dd7-813a-4e05-8045-3f33dbe02edc" containerName="kserve-container"
Sep 16 19:21:13.954039 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:13.953782    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="15187dd7-813a-4e05-8045-3f33dbe02edc" containerName="kube-rbac-proxy"
Sep 16 19:21:13.958485 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:13.958467    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"
Sep 16 19:21:13.966912 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:21:13.966881    2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"isvc-xgboost-predictor-7566d65bf7-7fvqz\" is forbidden: User \"system:node:ip-10-0-137-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-137-241.ec2.internal' and this object" podUID="feadf624-0953-494e-a4e4-925e17cc2043" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"
Sep 16 19:21:13.966912 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:21:13.966887    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"isvc-xgboost-kube-rbac-proxy-sar-config\" is forbidden: User \"system:node:ip-10-0-137-241.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-137-241.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\"" type="*v1.ConfigMap"
Sep 16 19:21:13.967087 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:21:13.966951    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"isvc-xgboost-predictor-serving-cert\" is forbidden: User \"system:node:ip-10-0-137-241.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kserve-ci-e2e-test\": no relationship found between node 'ip-10-0-137-241.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\"" type="*v1.Secret"
Sep 16 19:21:14.000793 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:14.000760    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"]
Sep 16 19:21:14.070930 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:14.070896    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhvrk\" (UniqueName: \"kubernetes.io/projected/feadf624-0953-494e-a4e4-925e17cc2043-kube-api-access-qhvrk\") pod \"isvc-xgboost-predictor-7566d65bf7-7fvqz\" (UID: \"feadf624-0953-494e-a4e4-925e17cc2043\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"
Sep 16 19:21:14.071132 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:14.070972    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feadf624-0953-494e-a4e4-925e17cc2043-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-7fvqz\" (UID: \"feadf624-0953-494e-a4e4-925e17cc2043\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"
Sep 16 19:21:14.071132 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:14.070998    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/feadf624-0953-494e-a4e4-925e17cc2043-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-7fvqz\" (UID: \"feadf624-0953-494e-a4e4-925e17cc2043\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"
Sep 16 19:21:14.071132 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:14.071104    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/feadf624-0953-494e-a4e4-925e17cc2043-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-7fvqz\" (UID: \"feadf624-0953-494e-a4e4-925e17cc2043\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"
Sep 16 19:21:14.071270 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:14.071144    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feadf624-0953-494e-a4e4-925e17cc2043-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-7fvqz\" (UID: \"feadf624-0953-494e-a4e4-925e17cc2043\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"
Sep 16 19:21:14.172603 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:14.172569    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feadf624-0953-494e-a4e4-925e17cc2043-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-7fvqz\" (UID: \"feadf624-0953-494e-a4e4-925e17cc2043\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"
Sep 16 19:21:14.172603 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:14.172604    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/feadf624-0953-494e-a4e4-925e17cc2043-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-7fvqz\" (UID: \"feadf624-0953-494e-a4e4-925e17cc2043\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"
Sep 16 19:21:14.172857 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:14.172658    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/feadf624-0953-494e-a4e4-925e17cc2043-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-7fvqz\" (UID: \"feadf624-0953-494e-a4e4-925e17cc2043\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"
Sep 16 19:21:14.172857 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:14.172679    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feadf624-0953-494e-a4e4-925e17cc2043-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-7fvqz\" (UID: \"feadf624-0953-494e-a4e4-925e17cc2043\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"
Sep 16 19:21:14.172857 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:14.172704    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhvrk\" (UniqueName: \"kubernetes.io/projected/feadf624-0953-494e-a4e4-925e17cc2043-kube-api-access-qhvrk\") pod \"isvc-xgboost-predictor-7566d65bf7-7fvqz\" (UID: \"feadf624-0953-494e-a4e4-925e17cc2043\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"
Sep 16 19:21:14.173133 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:14.173112    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/feadf624-0953-494e-a4e4-925e17cc2043-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-7fvqz\" (UID: \"feadf624-0953-494e-a4e4-925e17cc2043\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"
Sep 16 19:21:14.173464 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:14.173445    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/feadf624-0953-494e-a4e4-925e17cc2043-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-7fvqz\" (UID: \"feadf624-0953-494e-a4e4-925e17cc2043\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"
Sep 16 19:21:14.184984 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:14.184964    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhvrk\" (UniqueName: \"kubernetes.io/projected/feadf624-0953-494e-a4e4-925e17cc2043-kube-api-access-qhvrk\") pod \"isvc-xgboost-predictor-7566d65bf7-7fvqz\" (UID: \"feadf624-0953-494e-a4e4-925e17cc2043\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"
Sep 16 19:21:15.122826 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:15.122788    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Sep 16 19:21:15.123377 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:15.123356    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feadf624-0953-494e-a4e4-925e17cc2043-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-7fvqz\" (UID: \"feadf624-0953-494e-a4e4-925e17cc2043\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"
Sep 16 19:21:15.162708 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:15.162680    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Sep 16 19:21:15.165091 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:15.165070    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feadf624-0953-494e-a4e4-925e17cc2043-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-7fvqz\" (UID: \"feadf624-0953-494e-a4e4-925e17cc2043\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"
Sep 16 19:21:15.167929 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:15.167909    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"
Sep 16 19:21:15.449381 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:15.449298    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"]
Sep 16 19:21:15.455645 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:21:15.455620    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeadf624_0953_494e_a4e4_925e17cc2043.slice/crio-9f0025e67c6ac06232d95fff4b4728e95061792d912478b7086a14c08b2f4173 WatchSource:0}: Error finding container 9f0025e67c6ac06232d95fff4b4728e95061792d912478b7086a14c08b2f4173: Status 404 returned error can't find the container with id 9f0025e67c6ac06232d95fff4b4728e95061792d912478b7086a14c08b2f4173
Sep 16 19:21:16.101650 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:16.101609    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz" event={"ID":"feadf624-0953-494e-a4e4-925e17cc2043","Type":"ContainerStarted","Data":"3b3e0969ea2f1d5f0fded499de8c15cf1707cc9e051822b45044fb4955045c93"}
Sep 16 19:21:16.101879 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:16.101686    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz" event={"ID":"feadf624-0953-494e-a4e4-925e17cc2043","Type":"ContainerStarted","Data":"9f0025e67c6ac06232d95fff4b4728e95061792d912478b7086a14c08b2f4173"}
Sep 16 19:21:20.123134 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:20.123092    2554 generic.go:356] "Generic (PLEG): container finished" podID="feadf624-0953-494e-a4e4-925e17cc2043" containerID="3b3e0969ea2f1d5f0fded499de8c15cf1707cc9e051822b45044fb4955045c93" exitCode=0
Sep 16 19:21:20.123514 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:20.123143    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz" event={"ID":"feadf624-0953-494e-a4e4-925e17cc2043","Type":"ContainerDied","Data":"3b3e0969ea2f1d5f0fded499de8c15cf1707cc9e051822b45044fb4955045c93"}
Sep 16 19:21:37.226830 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:37.226749    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz" event={"ID":"feadf624-0953-494e-a4e4-925e17cc2043","Type":"ContainerStarted","Data":"27d824f81fb61d743e6dea3dcb3accb204494ba57661da265b736f4129f9807f"}
Sep 16 19:21:37.226830 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:37.226814    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz" event={"ID":"feadf624-0953-494e-a4e4-925e17cc2043","Type":"ContainerStarted","Data":"c4e63472c71eb7f43c86cfe9b712bfbee3d3cae63d157d88d4a62c80832fb30a"}
Sep 16 19:21:37.227261 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:37.227101    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"
Sep 16 19:21:37.253369 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:37.253320    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz" podStartSLOduration=3.6086109 podStartE2EDuration="24.253301004s" totalImagesPullingTime="16.695365816s" totalInitContainerRuntime="3.949324288s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:21:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:21:37.251373204 +0000 UTC m=+2207.327142838" watchObservedRunningTime="2026-09-16 19:21:37.253301004 +0000 UTC m=+2207.329070635"
Sep 16 19:21:38.232772 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:38.232739    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"
Sep 16 19:21:38.234144 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:38.234119    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz" podUID="feadf624-0953-494e-a4e4-925e17cc2043" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:21:39.237771 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:39.237733    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz" podUID="feadf624-0953-494e-a4e4-925e17cc2043" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:21:44.242090 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:44.242054    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"
Sep 16 19:21:44.242631 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:44.242601    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz" podUID="feadf624-0953-494e-a4e4-925e17cc2043" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:21:54.243569 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:21:54.243526    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz" podUID="feadf624-0953-494e-a4e4-925e17cc2043" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:22:04.242682 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:04.242640    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz" podUID="feadf624-0953-494e-a4e4-925e17cc2043" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:22:14.242517 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:14.242440    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz" podUID="feadf624-0953-494e-a4e4-925e17cc2043" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:22:24.243300 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:24.243257    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz" podUID="feadf624-0953-494e-a4e4-925e17cc2043" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:22:30.541231 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:30.541196    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"]
Sep 16 19:22:30.541660 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:30.541480    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn" podUID="2f686fd5-3880-4fe1-bd55-300065dc46e5" containerName="kserve-container" containerID="cri-o://d4a99091bef2e873875e8a5d187aa922ec71ed99bcb1849b340ed389fe1cdcca" gracePeriod=30
Sep 16 19:22:30.541786 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:30.541516    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn" podUID="2f686fd5-3880-4fe1-bd55-300065dc46e5" containerName="kube-rbac-proxy" containerID="cri-o://391d55008cbdcab42f36232f5168b67b5e726377215be1f36baf9fe634679889" gracePeriod=30
Sep 16 19:22:31.487736 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:31.487685    2554 generic.go:356] "Generic (PLEG): container finished" podID="2f686fd5-3880-4fe1-bd55-300065dc46e5" containerID="391d55008cbdcab42f36232f5168b67b5e726377215be1f36baf9fe634679889" exitCode=2
Sep 16 19:22:31.487899 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:31.487752    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn" event={"ID":"2f686fd5-3880-4fe1-bd55-300065dc46e5","Type":"ContainerDied","Data":"391d55008cbdcab42f36232f5168b67b5e726377215be1f36baf9fe634679889"}
Sep 16 19:22:32.438792 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:32.438744    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn" podUID="2f686fd5-3880-4fe1-bd55-300065dc46e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Sep 16 19:22:34.242579 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:34.242541    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz" podUID="feadf624-0953-494e-a4e4-925e17cc2043" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:22:34.636085 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:34.636043    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn" podUID="2f686fd5-3880-4fe1-bd55-300065dc46e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.52:8643/healthz\": dial tcp 10.134.0.52:8643: connect: connection refused"
Sep 16 19:22:35.090156 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.090127    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"
Sep 16 19:22:35.144213 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.144178    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f686fd5-3880-4fe1-bd55-300065dc46e5-kube-rbac-proxy-sar-config\") pod \"2f686fd5-3880-4fe1-bd55-300065dc46e5\" (UID: \"2f686fd5-3880-4fe1-bd55-300065dc46e5\") "
Sep 16 19:22:35.144213 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.144215    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f686fd5-3880-4fe1-bd55-300065dc46e5-proxy-tls\") pod \"2f686fd5-3880-4fe1-bd55-300065dc46e5\" (UID: \"2f686fd5-3880-4fe1-bd55-300065dc46e5\") "
Sep 16 19:22:35.144495 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.144296    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f686fd5-3880-4fe1-bd55-300065dc46e5-cabundle-cert\") pod \"2f686fd5-3880-4fe1-bd55-300065dc46e5\" (UID: \"2f686fd5-3880-4fe1-bd55-300065dc46e5\") "
Sep 16 19:22:35.144495 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.144348    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f686fd5-3880-4fe1-bd55-300065dc46e5-kserve-provision-location\") pod \"2f686fd5-3880-4fe1-bd55-300065dc46e5\" (UID: \"2f686fd5-3880-4fe1-bd55-300065dc46e5\") "
Sep 16 19:22:35.144495 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.144413    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdl8x\" (UniqueName: \"kubernetes.io/projected/2f686fd5-3880-4fe1-bd55-300065dc46e5-kube-api-access-cdl8x\") pod \"2f686fd5-3880-4fe1-bd55-300065dc46e5\" (UID: \"2f686fd5-3880-4fe1-bd55-300065dc46e5\") "
Sep 16 19:22:35.144887 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.144846    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f686fd5-3880-4fe1-bd55-300065dc46e5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2f686fd5-3880-4fe1-bd55-300065dc46e5" (UID: "2f686fd5-3880-4fe1-bd55-300065dc46e5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:22:35.145011 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.144981    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f686fd5-3880-4fe1-bd55-300065dc46e5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2f686fd5-3880-4fe1-bd55-300065dc46e5" (UID: "2f686fd5-3880-4fe1-bd55-300065dc46e5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:22:35.145073 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.145010    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f686fd5-3880-4fe1-bd55-300065dc46e5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2f686fd5-3880-4fe1-bd55-300065dc46e5" (UID: "2f686fd5-3880-4fe1-bd55-300065dc46e5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:35.147383 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.147346    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f686fd5-3880-4fe1-bd55-300065dc46e5-kube-api-access-cdl8x" (OuterVolumeSpecName: "kube-api-access-cdl8x") pod "2f686fd5-3880-4fe1-bd55-300065dc46e5" (UID: "2f686fd5-3880-4fe1-bd55-300065dc46e5"). InnerVolumeSpecName "kube-api-access-cdl8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:22:35.147507 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.147444    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f686fd5-3880-4fe1-bd55-300065dc46e5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2f686fd5-3880-4fe1-bd55-300065dc46e5" (UID: "2f686fd5-3880-4fe1-bd55-300065dc46e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:22:35.147507 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.145118    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2f686fd5-3880-4fe1-bd55-300065dc46e5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:22:35.147507 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.147503    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2f686fd5-3880-4fe1-bd55-300065dc46e5-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:22:35.147668 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.147518    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2f686fd5-3880-4fe1-bd55-300065dc46e5-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:22:35.248473 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.248446    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2f686fd5-3880-4fe1-bd55-300065dc46e5-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:22:35.248473 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.248474    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdl8x\" (UniqueName: \"kubernetes.io/projected/2f686fd5-3880-4fe1-bd55-300065dc46e5-kube-api-access-cdl8x\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:22:35.507632 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.507542    2554 generic.go:356] "Generic (PLEG): container finished" podID="2f686fd5-3880-4fe1-bd55-300065dc46e5" containerID="d4a99091bef2e873875e8a5d187aa922ec71ed99bcb1849b340ed389fe1cdcca" exitCode=0
Sep 16 19:22:35.507632 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.507610    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"
Sep 16 19:22:35.507848 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.507634    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn" event={"ID":"2f686fd5-3880-4fe1-bd55-300065dc46e5","Type":"ContainerDied","Data":"d4a99091bef2e873875e8a5d187aa922ec71ed99bcb1849b340ed389fe1cdcca"}
Sep 16 19:22:35.507848 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.507700    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn" event={"ID":"2f686fd5-3880-4fe1-bd55-300065dc46e5","Type":"ContainerDied","Data":"c38518d0151aeef578e01da31e356eed2d22711488b21f3a900e6d1f3070f56d"}
Sep 16 19:22:35.507848 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.507744    2554 scope.go:117] "RemoveContainer" containerID="391d55008cbdcab42f36232f5168b67b5e726377215be1f36baf9fe634679889"
Sep 16 19:22:35.516533 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.516516    2554 scope.go:117] "RemoveContainer" containerID="d4a99091bef2e873875e8a5d187aa922ec71ed99bcb1849b340ed389fe1cdcca"
Sep 16 19:22:35.523816 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.523796    2554 scope.go:117] "RemoveContainer" containerID="c99b04df88b60cf0f3647490a7c3d2f705fbde00ccda5f955e93578e6b880010"
Sep 16 19:22:35.530845 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.530824    2554 scope.go:117] "RemoveContainer" containerID="391d55008cbdcab42f36232f5168b67b5e726377215be1f36baf9fe634679889"
Sep 16 19:22:35.531093 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:22:35.531077    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"391d55008cbdcab42f36232f5168b67b5e726377215be1f36baf9fe634679889\": container with ID starting with 391d55008cbdcab42f36232f5168b67b5e726377215be1f36baf9fe634679889 not found: ID does not exist" containerID="391d55008cbdcab42f36232f5168b67b5e726377215be1f36baf9fe634679889"
Sep 16 19:22:35.531187 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.531100    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"391d55008cbdcab42f36232f5168b67b5e726377215be1f36baf9fe634679889"} err="failed to get container status \"391d55008cbdcab42f36232f5168b67b5e726377215be1f36baf9fe634679889\": rpc error: code = NotFound desc = could not find container \"391d55008cbdcab42f36232f5168b67b5e726377215be1f36baf9fe634679889\": container with ID starting with 391d55008cbdcab42f36232f5168b67b5e726377215be1f36baf9fe634679889 not found: ID does not exist"
Sep 16 19:22:35.531187 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.531122    2554 scope.go:117] "RemoveContainer" containerID="d4a99091bef2e873875e8a5d187aa922ec71ed99bcb1849b340ed389fe1cdcca"
Sep 16 19:22:35.531416 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:22:35.531380    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4a99091bef2e873875e8a5d187aa922ec71ed99bcb1849b340ed389fe1cdcca\": container with ID starting with d4a99091bef2e873875e8a5d187aa922ec71ed99bcb1849b340ed389fe1cdcca not found: ID does not exist" containerID="d4a99091bef2e873875e8a5d187aa922ec71ed99bcb1849b340ed389fe1cdcca"
Sep 16 19:22:35.531508 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.531409    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a99091bef2e873875e8a5d187aa922ec71ed99bcb1849b340ed389fe1cdcca"} err="failed to get container status \"d4a99091bef2e873875e8a5d187aa922ec71ed99bcb1849b340ed389fe1cdcca\": rpc error: code = NotFound desc = could not find container \"d4a99091bef2e873875e8a5d187aa922ec71ed99bcb1849b340ed389fe1cdcca\": container with ID starting with d4a99091bef2e873875e8a5d187aa922ec71ed99bcb1849b340ed389fe1cdcca not found: ID does not exist"
Sep 16 19:22:35.531508 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.531430    2554 scope.go:117] "RemoveContainer" containerID="c99b04df88b60cf0f3647490a7c3d2f705fbde00ccda5f955e93578e6b880010"
Sep 16 19:22:35.531796 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:22:35.531773    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c99b04df88b60cf0f3647490a7c3d2f705fbde00ccda5f955e93578e6b880010\": container with ID starting with c99b04df88b60cf0f3647490a7c3d2f705fbde00ccda5f955e93578e6b880010 not found: ID does not exist" containerID="c99b04df88b60cf0f3647490a7c3d2f705fbde00ccda5f955e93578e6b880010"
Sep 16 19:22:35.531882 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.531803    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c99b04df88b60cf0f3647490a7c3d2f705fbde00ccda5f955e93578e6b880010"} err="failed to get container status \"c99b04df88b60cf0f3647490a7c3d2f705fbde00ccda5f955e93578e6b880010\": rpc error: code = NotFound desc = could not find container \"c99b04df88b60cf0f3647490a7c3d2f705fbde00ccda5f955e93578e6b880010\": container with ID starting with c99b04df88b60cf0f3647490a7c3d2f705fbde00ccda5f955e93578e6b880010 not found: ID does not exist"
Sep 16 19:22:35.533875 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.533857    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"]
Sep 16 19:22:35.537680 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:35.537662    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-857b7ffb9f-dtbhn"]
Sep 16 19:22:36.034962 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.034929    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf"]
Sep 16 19:22:36.035395 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.035376    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f686fd5-3880-4fe1-bd55-300065dc46e5" containerName="kube-rbac-proxy"
Sep 16 19:22:36.035490 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.035398    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f686fd5-3880-4fe1-bd55-300065dc46e5" containerName="kube-rbac-proxy"
Sep 16 19:22:36.035490 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.035420    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f686fd5-3880-4fe1-bd55-300065dc46e5" containerName="kserve-container"
Sep 16 19:22:36.035490 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.035428    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f686fd5-3880-4fe1-bd55-300065dc46e5" containerName="kserve-container"
Sep 16 19:22:36.035490 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.035448    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f686fd5-3880-4fe1-bd55-300065dc46e5" containerName="storage-initializer"
Sep 16 19:22:36.035490 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.035458    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f686fd5-3880-4fe1-bd55-300065dc46e5" containerName="storage-initializer"
Sep 16 19:22:36.035788 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.035543    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f686fd5-3880-4fe1-bd55-300065dc46e5" containerName="kserve-container"
Sep 16 19:22:36.035788 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.035555    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f686fd5-3880-4fe1-bd55-300065dc46e5" containerName="kube-rbac-proxy"
Sep 16 19:22:36.039009 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.038979    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf"
Sep 16 19:22:36.042834 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.042816    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 16 19:22:36.043197 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.043182    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 16 19:22:36.053656 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.053632    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/417f90db-905b-45bd-8cb6-80bb7c7410c8-kserve-provision-location\") pod \"isvc-canary-stop-predictor-57d5c586ff-2hbzf\" (UID: \"417f90db-905b-45bd-8cb6-80bb7c7410c8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf"
Sep 16 19:22:36.053784 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.053676    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/417f90db-905b-45bd-8cb6-80bb7c7410c8-proxy-tls\") pod \"isvc-canary-stop-predictor-57d5c586ff-2hbzf\" (UID: \"417f90db-905b-45bd-8cb6-80bb7c7410c8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf"
Sep 16 19:22:36.053784 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.053698    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/417f90db-905b-45bd-8cb6-80bb7c7410c8-cabundle-cert\") pod \"isvc-canary-stop-predictor-57d5c586ff-2hbzf\" (UID: \"417f90db-905b-45bd-8cb6-80bb7c7410c8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf"
Sep 16 19:22:36.053885 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.053786    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddlkk\" (UniqueName: \"kubernetes.io/projected/417f90db-905b-45bd-8cb6-80bb7c7410c8-kube-api-access-ddlkk\") pod \"isvc-canary-stop-predictor-57d5c586ff-2hbzf\" (UID: \"417f90db-905b-45bd-8cb6-80bb7c7410c8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf"
Sep 16 19:22:36.053885 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.053813    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/417f90db-905b-45bd-8cb6-80bb7c7410c8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-57d5c586ff-2hbzf\" (UID: \"417f90db-905b-45bd-8cb6-80bb7c7410c8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf"
Sep 16 19:22:36.058256 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.058231    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf"]
Sep 16 19:22:36.154612 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.154575    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/417f90db-905b-45bd-8cb6-80bb7c7410c8-proxy-tls\") pod \"isvc-canary-stop-predictor-57d5c586ff-2hbzf\" (UID: \"417f90db-905b-45bd-8cb6-80bb7c7410c8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf"
Sep 16 19:22:36.154612 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.154617    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/417f90db-905b-45bd-8cb6-80bb7c7410c8-cabundle-cert\") pod \"isvc-canary-stop-predictor-57d5c586ff-2hbzf\" (UID: \"417f90db-905b-45bd-8cb6-80bb7c7410c8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf"
Sep 16 19:22:36.154899 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.154645    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddlkk\" (UniqueName: \"kubernetes.io/projected/417f90db-905b-45bd-8cb6-80bb7c7410c8-kube-api-access-ddlkk\") pod \"isvc-canary-stop-predictor-57d5c586ff-2hbzf\" (UID: \"417f90db-905b-45bd-8cb6-80bb7c7410c8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf"
Sep 16 19:22:36.154899 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.154666    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/417f90db-905b-45bd-8cb6-80bb7c7410c8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-57d5c586ff-2hbzf\" (UID: \"417f90db-905b-45bd-8cb6-80bb7c7410c8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf"
Sep 16 19:22:36.154899 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.154745    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/417f90db-905b-45bd-8cb6-80bb7c7410c8-kserve-provision-location\") pod \"isvc-canary-stop-predictor-57d5c586ff-2hbzf\" (UID: \"417f90db-905b-45bd-8cb6-80bb7c7410c8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf"
Sep 16 19:22:36.155172 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.155133    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/417f90db-905b-45bd-8cb6-80bb7c7410c8-kserve-provision-location\") pod \"isvc-canary-stop-predictor-57d5c586ff-2hbzf\" (UID: \"417f90db-905b-45bd-8cb6-80bb7c7410c8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf"
Sep 16 19:22:36.155335 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.155316    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/417f90db-905b-45bd-8cb6-80bb7c7410c8-cabundle-cert\") pod \"isvc-canary-stop-predictor-57d5c586ff-2hbzf\" (UID: \"417f90db-905b-45bd-8cb6-80bb7c7410c8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf"
Sep 16 19:22:36.155422 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.155403    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/417f90db-905b-45bd-8cb6-80bb7c7410c8-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-57d5c586ff-2hbzf\" (UID: \"417f90db-905b-45bd-8cb6-80bb7c7410c8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf"
Sep 16 19:22:36.157061 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.157045    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/417f90db-905b-45bd-8cb6-80bb7c7410c8-proxy-tls\") pod \"isvc-canary-stop-predictor-57d5c586ff-2hbzf\" (UID: \"417f90db-905b-45bd-8cb6-80bb7c7410c8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf"
Sep 16 19:22:36.164525 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.164504    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddlkk\" (UniqueName: \"kubernetes.io/projected/417f90db-905b-45bd-8cb6-80bb7c7410c8-kube-api-access-ddlkk\") pod \"isvc-canary-stop-predictor-57d5c586ff-2hbzf\" (UID: \"417f90db-905b-45bd-8cb6-80bb7c7410c8\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf"
Sep 16 19:22:36.349673 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.349599    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf"
Sep 16 19:22:36.443319 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.443287    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f686fd5-3880-4fe1-bd55-300065dc46e5" path="/var/lib/kubelet/pods/2f686fd5-3880-4fe1-bd55-300065dc46e5/volumes"
Sep 16 19:22:36.632383 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:36.632353    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf"]
Sep 16 19:22:36.635060 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:22:36.635031    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417f90db_905b_45bd_8cb6_80bb7c7410c8.slice/crio-44639d92b39d2a897c56adbfb94e040a9cbd49f7761bd6ca891046cc0cb1695c WatchSource:0}: Error finding container 44639d92b39d2a897c56adbfb94e040a9cbd49f7761bd6ca891046cc0cb1695c: Status 404 returned error can't find the container with id 44639d92b39d2a897c56adbfb94e040a9cbd49f7761bd6ca891046cc0cb1695c
Sep 16 19:22:37.519599 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:37.519562    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf" event={"ID":"417f90db-905b-45bd-8cb6-80bb7c7410c8","Type":"ContainerStarted","Data":"756c845dc6d5cd7b564d532a9c2fc3664cea4d54184a0d4be9b54d324d53bec6"}
Sep 16 19:22:37.520037 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:37.519622    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf" event={"ID":"417f90db-905b-45bd-8cb6-80bb7c7410c8","Type":"ContainerStarted","Data":"44639d92b39d2a897c56adbfb94e040a9cbd49f7761bd6ca891046cc0cb1695c"}
Sep 16 19:22:41.540474 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:41.540440    2554 generic.go:356] "Generic (PLEG): container finished" podID="417f90db-905b-45bd-8cb6-80bb7c7410c8" containerID="756c845dc6d5cd7b564d532a9c2fc3664cea4d54184a0d4be9b54d324d53bec6" exitCode=0
Sep 16 19:22:41.540876 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:41.540515    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf" event={"ID":"417f90db-905b-45bd-8cb6-80bb7c7410c8","Type":"ContainerDied","Data":"756c845dc6d5cd7b564d532a9c2fc3664cea4d54184a0d4be9b54d324d53bec6"}
Sep 16 19:22:42.546849 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:42.546808    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf" event={"ID":"417f90db-905b-45bd-8cb6-80bb7c7410c8","Type":"ContainerStarted","Data":"c7b77f19e4d17a8ef17ae0ff4b154cfaac0fa717c013669e432517189e3d740d"}
Sep 16 19:22:42.547224 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:42.546878    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf" event={"ID":"417f90db-905b-45bd-8cb6-80bb7c7410c8","Type":"ContainerStarted","Data":"10750c19050a6211b6c17d8cd066e29f335e2fa58d8263d5b7e0fae1280ceceb"}
Sep 16 19:22:42.547224 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:42.547058    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf"
Sep 16 19:22:42.566965 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:42.566920    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf" podStartSLOduration=3.723861489 podStartE2EDuration="7.566906919s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.84304543s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:22:42.565689828 +0000 UTC m=+2272.641459461" watchObservedRunningTime="2026-09-16 19:22:42.566906919 +0000 UTC m=+2272.642676621"
Sep 16 19:22:43.551242 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:43.551202    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf"
Sep 16 19:22:43.552606 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:43.552572    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf" podUID="417f90db-905b-45bd-8cb6-80bb7c7410c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:22:44.243913 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:44.243883    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"
Sep 16 19:22:44.555659 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:44.555621    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf" podUID="417f90db-905b-45bd-8cb6-80bb7c7410c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:22:49.561115 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:49.561088    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf"
Sep 16 19:22:49.561748 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:49.561709    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf" podUID="417f90db-905b-45bd-8cb6-80bb7c7410c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:22:54.032483 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:54.032446    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"]
Sep 16 19:22:54.032876 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:54.032688    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz" podUID="feadf624-0953-494e-a4e4-925e17cc2043" containerName="kserve-container" containerID="cri-o://c4e63472c71eb7f43c86cfe9b712bfbee3d3cae63d157d88d4a62c80832fb30a" gracePeriod=30
Sep 16 19:22:54.032876 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:54.032760    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz" podUID="feadf624-0953-494e-a4e4-925e17cc2043" containerName="kube-rbac-proxy" containerID="cri-o://27d824f81fb61d743e6dea3dcb3accb204494ba57661da265b736f4129f9807f" gracePeriod=30
Sep 16 19:22:54.238964 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:54.238919    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz" podUID="feadf624-0953-494e-a4e4-925e17cc2043" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Sep 16 19:22:54.242689 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:54.242660    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz" podUID="feadf624-0953-494e-a4e4-925e17cc2043" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Sep 16 19:22:54.601875 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:54.601843    2554 generic.go:356] "Generic (PLEG): container finished" podID="feadf624-0953-494e-a4e4-925e17cc2043" containerID="27d824f81fb61d743e6dea3dcb3accb204494ba57661da265b736f4129f9807f" exitCode=2
Sep 16 19:22:54.602037 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:54.601920    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz" event={"ID":"feadf624-0953-494e-a4e4-925e17cc2043","Type":"ContainerDied","Data":"27d824f81fb61d743e6dea3dcb3accb204494ba57661da265b736f4129f9807f"}
Sep 16 19:22:57.477129 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.477106    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"
Sep 16 19:22:57.553127 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.553054    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/feadf624-0953-494e-a4e4-925e17cc2043-cabundle-cert\") pod \"feadf624-0953-494e-a4e4-925e17cc2043\" (UID: \"feadf624-0953-494e-a4e4-925e17cc2043\") "
Sep 16 19:22:57.553293 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.553162    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feadf624-0953-494e-a4e4-925e17cc2043-kube-rbac-proxy-sar-config\") pod \"feadf624-0953-494e-a4e4-925e17cc2043\" (UID: \"feadf624-0953-494e-a4e4-925e17cc2043\") "
Sep 16 19:22:57.553293 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.553209    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feadf624-0953-494e-a4e4-925e17cc2043-proxy-tls\") pod \"feadf624-0953-494e-a4e4-925e17cc2043\" (UID: \"feadf624-0953-494e-a4e4-925e17cc2043\") "
Sep 16 19:22:57.553293 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.553239    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/feadf624-0953-494e-a4e4-925e17cc2043-kserve-provision-location\") pod \"feadf624-0953-494e-a4e4-925e17cc2043\" (UID: \"feadf624-0953-494e-a4e4-925e17cc2043\") "
Sep 16 19:22:57.553293 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.553276    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhvrk\" (UniqueName: \"kubernetes.io/projected/feadf624-0953-494e-a4e4-925e17cc2043-kube-api-access-qhvrk\") pod \"feadf624-0953-494e-a4e4-925e17cc2043\" (UID: \"feadf624-0953-494e-a4e4-925e17cc2043\") "
Sep 16 19:22:57.553576 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.553454    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feadf624-0953-494e-a4e4-925e17cc2043-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "feadf624-0953-494e-a4e4-925e17cc2043" (UID: "feadf624-0953-494e-a4e4-925e17cc2043"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:22:57.553751 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.553613    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feadf624-0953-494e-a4e4-925e17cc2043-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "feadf624-0953-494e-a4e4-925e17cc2043" (UID: "feadf624-0953-494e-a4e4-925e17cc2043"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:22:57.553831 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.553805    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feadf624-0953-494e-a4e4-925e17cc2043-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "feadf624-0953-494e-a4e4-925e17cc2043" (UID: "feadf624-0953-494e-a4e4-925e17cc2043"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:22:57.563598 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.563554    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/feadf624-0953-494e-a4e4-925e17cc2043-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "feadf624-0953-494e-a4e4-925e17cc2043" (UID: "feadf624-0953-494e-a4e4-925e17cc2043"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:22:57.564033 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.564013    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feadf624-0953-494e-a4e4-925e17cc2043-kube-api-access-qhvrk" (OuterVolumeSpecName: "kube-api-access-qhvrk") pod "feadf624-0953-494e-a4e4-925e17cc2043" (UID: "feadf624-0953-494e-a4e4-925e17cc2043"). InnerVolumeSpecName "kube-api-access-qhvrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:22:57.616456 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.616428    2554 generic.go:356] "Generic (PLEG): container finished" podID="feadf624-0953-494e-a4e4-925e17cc2043" containerID="c4e63472c71eb7f43c86cfe9b712bfbee3d3cae63d157d88d4a62c80832fb30a" exitCode=0
Sep 16 19:22:57.616562 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.616505    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"
Sep 16 19:22:57.616562 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.616519    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz" event={"ID":"feadf624-0953-494e-a4e4-925e17cc2043","Type":"ContainerDied","Data":"c4e63472c71eb7f43c86cfe9b712bfbee3d3cae63d157d88d4a62c80832fb30a"}
Sep 16 19:22:57.616689 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.616601    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz" event={"ID":"feadf624-0953-494e-a4e4-925e17cc2043","Type":"ContainerDied","Data":"9f0025e67c6ac06232d95fff4b4728e95061792d912478b7086a14c08b2f4173"}
Sep 16 19:22:57.616689 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.616629    2554 scope.go:117] "RemoveContainer" containerID="27d824f81fb61d743e6dea3dcb3accb204494ba57661da265b736f4129f9807f"
Sep 16 19:22:57.624933 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.624919    2554 scope.go:117] "RemoveContainer" containerID="c4e63472c71eb7f43c86cfe9b712bfbee3d3cae63d157d88d4a62c80832fb30a"
Sep 16 19:22:57.632215 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.632199    2554 scope.go:117] "RemoveContainer" containerID="3b3e0969ea2f1d5f0fded499de8c15cf1707cc9e051822b45044fb4955045c93"
Sep 16 19:22:57.640046 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.639834    2554 scope.go:117] "RemoveContainer" containerID="27d824f81fb61d743e6dea3dcb3accb204494ba57661da265b736f4129f9807f"
Sep 16 19:22:57.640046 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.639856    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"]
Sep 16 19:22:57.640383 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:22:57.640323    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27d824f81fb61d743e6dea3dcb3accb204494ba57661da265b736f4129f9807f\": container with ID starting with 27d824f81fb61d743e6dea3dcb3accb204494ba57661da265b736f4129f9807f not found: ID does not exist" containerID="27d824f81fb61d743e6dea3dcb3accb204494ba57661da265b736f4129f9807f"
Sep 16 19:22:57.640383 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.640357    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27d824f81fb61d743e6dea3dcb3accb204494ba57661da265b736f4129f9807f"} err="failed to get container status \"27d824f81fb61d743e6dea3dcb3accb204494ba57661da265b736f4129f9807f\": rpc error: code = NotFound desc = could not find container \"27d824f81fb61d743e6dea3dcb3accb204494ba57661da265b736f4129f9807f\": container with ID starting with 27d824f81fb61d743e6dea3dcb3accb204494ba57661da265b736f4129f9807f not found: ID does not exist"
Sep 16 19:22:57.640383 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.640379    2554 scope.go:117] "RemoveContainer" containerID="c4e63472c71eb7f43c86cfe9b712bfbee3d3cae63d157d88d4a62c80832fb30a"
Sep 16 19:22:57.640627 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:22:57.640611    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4e63472c71eb7f43c86cfe9b712bfbee3d3cae63d157d88d4a62c80832fb30a\": container with ID starting with c4e63472c71eb7f43c86cfe9b712bfbee3d3cae63d157d88d4a62c80832fb30a not found: ID does not exist" containerID="c4e63472c71eb7f43c86cfe9b712bfbee3d3cae63d157d88d4a62c80832fb30a"
Sep 16 19:22:57.640688 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.640629    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4e63472c71eb7f43c86cfe9b712bfbee3d3cae63d157d88d4a62c80832fb30a"} err="failed to get container status \"c4e63472c71eb7f43c86cfe9b712bfbee3d3cae63d157d88d4a62c80832fb30a\": rpc error: code = NotFound desc = could not find container \"c4e63472c71eb7f43c86cfe9b712bfbee3d3cae63d157d88d4a62c80832fb30a\": container with ID starting with c4e63472c71eb7f43c86cfe9b712bfbee3d3cae63d157d88d4a62c80832fb30a not found: ID does not exist"
Sep 16 19:22:57.640688 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.640641    2554 scope.go:117] "RemoveContainer" containerID="3b3e0969ea2f1d5f0fded499de8c15cf1707cc9e051822b45044fb4955045c93"
Sep 16 19:22:57.640877 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:22:57.640861    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b3e0969ea2f1d5f0fded499de8c15cf1707cc9e051822b45044fb4955045c93\": container with ID starting with 3b3e0969ea2f1d5f0fded499de8c15cf1707cc9e051822b45044fb4955045c93 not found: ID does not exist" containerID="3b3e0969ea2f1d5f0fded499de8c15cf1707cc9e051822b45044fb4955045c93"
Sep 16 19:22:57.640939 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.640880    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b3e0969ea2f1d5f0fded499de8c15cf1707cc9e051822b45044fb4955045c93"} err="failed to get container status \"3b3e0969ea2f1d5f0fded499de8c15cf1707cc9e051822b45044fb4955045c93\": rpc error: code = NotFound desc = could not find container \"3b3e0969ea2f1d5f0fded499de8c15cf1707cc9e051822b45044fb4955045c93\": container with ID starting with 3b3e0969ea2f1d5f0fded499de8c15cf1707cc9e051822b45044fb4955045c93 not found: ID does not exist"
Sep 16 19:22:57.642314 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.642294    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-7fvqz"]
Sep 16 19:22:57.654409 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.654388    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/feadf624-0953-494e-a4e4-925e17cc2043-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:22:57.654409 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.654407    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/feadf624-0953-494e-a4e4-925e17cc2043-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:22:57.654571 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.654417    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/feadf624-0953-494e-a4e4-925e17cc2043-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:22:57.654571 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.654425    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhvrk\" (UniqueName: \"kubernetes.io/projected/feadf624-0953-494e-a4e4-925e17cc2043-kube-api-access-qhvrk\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:22:57.654571 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:57.654445    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/feadf624-0953-494e-a4e4-925e17cc2043-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:22:58.443758 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:58.443707    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="feadf624-0953-494e-a4e4-925e17cc2043" path="/var/lib/kubelet/pods/feadf624-0953-494e-a4e4-925e17cc2043/volumes"
Sep 16 19:22:59.212130 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.212100    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98"]
Sep 16 19:22:59.212616 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.212596    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feadf624-0953-494e-a4e4-925e17cc2043" containerName="kserve-container"
Sep 16 19:22:59.212699 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.212619    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="feadf624-0953-494e-a4e4-925e17cc2043" containerName="kserve-container"
Sep 16 19:22:59.212699 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.212633    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feadf624-0953-494e-a4e4-925e17cc2043" containerName="storage-initializer"
Sep 16 19:22:59.212699 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.212641    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="feadf624-0953-494e-a4e4-925e17cc2043" containerName="storage-initializer"
Sep 16 19:22:59.212699 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.212655    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="feadf624-0953-494e-a4e4-925e17cc2043" containerName="kube-rbac-proxy"
Sep 16 19:22:59.212699 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.212663    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="feadf624-0953-494e-a4e4-925e17cc2043" containerName="kube-rbac-proxy"
Sep 16 19:22:59.213012 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.212795    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="feadf624-0953-494e-a4e4-925e17cc2043" containerName="kserve-container"
Sep 16 19:22:59.213012 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.212810    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="feadf624-0953-494e-a4e4-925e17cc2043" containerName="kube-rbac-proxy"
Sep 16 19:22:59.215217 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.215196    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98"
Sep 16 19:22:59.218731 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.218698    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Sep 16 19:22:59.218825 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.218747    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 16 19:22:59.230143 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.230121    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98"]
Sep 16 19:22:59.369908 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.369876    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6kzj\" (UniqueName: \"kubernetes.io/projected/0bf1ae1a-b155-4603-af80-eb136de46752-kube-api-access-b6kzj\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98\" (UID: \"0bf1ae1a-b155-4603-af80-eb136de46752\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98"
Sep 16 19:22:59.370076 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.369941    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bf1ae1a-b155-4603-af80-eb136de46752-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98\" (UID: \"0bf1ae1a-b155-4603-af80-eb136de46752\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98"
Sep 16 19:22:59.370076 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.370047    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bf1ae1a-b155-4603-af80-eb136de46752-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98\" (UID: \"0bf1ae1a-b155-4603-af80-eb136de46752\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98"
Sep 16 19:22:59.370159 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.370076    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bf1ae1a-b155-4603-af80-eb136de46752-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98\" (UID: \"0bf1ae1a-b155-4603-af80-eb136de46752\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98"
Sep 16 19:22:59.370159 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.370097    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bf1ae1a-b155-4603-af80-eb136de46752-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98\" (UID: \"0bf1ae1a-b155-4603-af80-eb136de46752\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98"
Sep 16 19:22:59.471205 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.471137    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bf1ae1a-b155-4603-af80-eb136de46752-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98\" (UID: \"0bf1ae1a-b155-4603-af80-eb136de46752\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98"
Sep 16 19:22:59.471205 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.471174    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bf1ae1a-b155-4603-af80-eb136de46752-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98\" (UID: \"0bf1ae1a-b155-4603-af80-eb136de46752\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98"
Sep 16 19:22:59.471205 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.471200    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bf1ae1a-b155-4603-af80-eb136de46752-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98\" (UID: \"0bf1ae1a-b155-4603-af80-eb136de46752\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98"
Sep 16 19:22:59.471438 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.471241    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6kzj\" (UniqueName: \"kubernetes.io/projected/0bf1ae1a-b155-4603-af80-eb136de46752-kube-api-access-b6kzj\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98\" (UID: \"0bf1ae1a-b155-4603-af80-eb136de46752\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98"
Sep 16 19:22:59.471438 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.471290    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bf1ae1a-b155-4603-af80-eb136de46752-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98\" (UID: \"0bf1ae1a-b155-4603-af80-eb136de46752\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98"
Sep 16 19:22:59.471640 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.471612    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bf1ae1a-b155-4603-af80-eb136de46752-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98\" (UID: \"0bf1ae1a-b155-4603-af80-eb136de46752\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98"
Sep 16 19:22:59.471831 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.471814    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bf1ae1a-b155-4603-af80-eb136de46752-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98\" (UID: \"0bf1ae1a-b155-4603-af80-eb136de46752\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98"
Sep 16 19:22:59.471938 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.471920    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bf1ae1a-b155-4603-af80-eb136de46752-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98\" (UID: \"0bf1ae1a-b155-4603-af80-eb136de46752\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98"
Sep 16 19:22:59.473651 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.473636    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bf1ae1a-b155-4603-af80-eb136de46752-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98\" (UID: \"0bf1ae1a-b155-4603-af80-eb136de46752\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98"
Sep 16 19:22:59.484841 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.484823    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6kzj\" (UniqueName: \"kubernetes.io/projected/0bf1ae1a-b155-4603-af80-eb136de46752-kube-api-access-b6kzj\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98\" (UID: \"0bf1ae1a-b155-4603-af80-eb136de46752\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98"
Sep 16 19:22:59.524740 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.524703    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98"
Sep 16 19:22:59.562374 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.562339    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf" podUID="417f90db-905b-45bd-8cb6-80bb7c7410c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:22:59.799482 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:22:59.799455    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98"]
Sep 16 19:22:59.801168 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:22:59.801144    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf1ae1a_b155_4603_af80_eb136de46752.slice/crio-8f4130075abb66798f4dbeb47911035adc2d068aeb9f259d2fbe211f1999e13c WatchSource:0}: Error finding container 8f4130075abb66798f4dbeb47911035adc2d068aeb9f259d2fbe211f1999e13c: Status 404 returned error can't find the container with id 8f4130075abb66798f4dbeb47911035adc2d068aeb9f259d2fbe211f1999e13c
Sep 16 19:23:00.633397 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:00.633368    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98" event={"ID":"0bf1ae1a-b155-4603-af80-eb136de46752","Type":"ContainerStarted","Data":"39b5d63cdf8925615ad4c95a82021efe0e5059359f420d78635a7cc181f4b9eb"}
Sep 16 19:23:00.633764 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:00.633423    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98" event={"ID":"0bf1ae1a-b155-4603-af80-eb136de46752","Type":"ContainerStarted","Data":"8f4130075abb66798f4dbeb47911035adc2d068aeb9f259d2fbe211f1999e13c"}
Sep 16 19:23:03.649042 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:03.648985    2554 generic.go:356] "Generic (PLEG): container finished" podID="0bf1ae1a-b155-4603-af80-eb136de46752" containerID="39b5d63cdf8925615ad4c95a82021efe0e5059359f420d78635a7cc181f4b9eb" exitCode=0
Sep 16 19:23:03.649528 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:03.649166    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98" event={"ID":"0bf1ae1a-b155-4603-af80-eb136de46752","Type":"ContainerDied","Data":"39b5d63cdf8925615ad4c95a82021efe0e5059359f420d78635a7cc181f4b9eb"}
Sep 16 19:23:04.656863 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:04.656818    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98" event={"ID":"0bf1ae1a-b155-4603-af80-eb136de46752","Type":"ContainerStarted","Data":"2d5db325fd1c631a958ae1e4a0448b54f65a80b0d2d810b24fa3c67bdc7e6697"}
Sep 16 19:23:04.657267 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:04.656902    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98" event={"ID":"0bf1ae1a-b155-4603-af80-eb136de46752","Type":"ContainerStarted","Data":"ec9f338f4449847357993cdbea69b213b02288bf2ed6ab36af9d1e680bff3e72"}
Sep 16 19:23:04.657267 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:04.657126    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98"
Sep 16 19:23:04.681918 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:04.681877    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98" podStartSLOduration=2.106063776 podStartE2EDuration="5.681862299s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.575798523s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:22:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:23:04.681203475 +0000 UTC m=+2294.756973118" watchObservedRunningTime="2026-09-16 19:23:04.681862299 +0000 UTC m=+2294.757631931"
Sep 16 19:23:05.662054 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:05.662022    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98"
Sep 16 19:23:09.562254 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:09.562219    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf" podUID="417f90db-905b-45bd-8cb6-80bb7c7410c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:23:11.672410 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:11.672384    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98"
Sep 16 19:23:19.562385 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:19.562348    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf" podUID="417f90db-905b-45bd-8cb6-80bb7c7410c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:23:29.561781 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:29.561745    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf" podUID="417f90db-905b-45bd-8cb6-80bb7c7410c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:23:39.562181 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:39.562146    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf" podUID="417f90db-905b-45bd-8cb6-80bb7c7410c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:23:41.676978 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:41.676952    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98"
Sep 16 19:23:49.333418 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:49.333344    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98"]
Sep 16 19:23:49.333795 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:49.333564    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98" podUID="0bf1ae1a-b155-4603-af80-eb136de46752" containerName="kserve-container" containerID="cri-o://ec9f338f4449847357993cdbea69b213b02288bf2ed6ab36af9d1e680bff3e72" gracePeriod=30
Sep 16 19:23:49.333795 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:49.333587    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98" podUID="0bf1ae1a-b155-4603-af80-eb136de46752" containerName="kube-rbac-proxy" containerID="cri-o://2d5db325fd1c631a958ae1e4a0448b54f65a80b0d2d810b24fa3c67bdc7e6697" gracePeriod=30
Sep 16 19:23:49.562364 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:49.562328    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf" podUID="417f90db-905b-45bd-8cb6-80bb7c7410c8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Sep 16 19:23:49.882170 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:49.882140    2554 generic.go:356] "Generic (PLEG): container finished" podID="0bf1ae1a-b155-4603-af80-eb136de46752" containerID="2d5db325fd1c631a958ae1e4a0448b54f65a80b0d2d810b24fa3c67bdc7e6697" exitCode=2
Sep 16 19:23:49.882333 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:49.882189    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98" event={"ID":"0bf1ae1a-b155-4603-af80-eb136de46752","Type":"ContainerDied","Data":"2d5db325fd1c631a958ae1e4a0448b54f65a80b0d2d810b24fa3c67bdc7e6697"}
Sep 16 19:23:51.668134 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:51.668090    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98" podUID="0bf1ae1a-b155-4603-af80-eb136de46752" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
Sep 16 19:23:51.673093 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:51.673066    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98" podUID="0bf1ae1a-b155-4603-af80-eb136de46752" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.55:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.134.0.55:8080: connect: connection refused"
Sep 16 19:23:54.464238 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.464207    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt"]
Sep 16 19:23:54.466795 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.466777    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt"
Sep 16 19:23:54.469001 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.468979    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Sep 16 19:23:54.469001 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.468998    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Sep 16 19:23:54.480264 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.480243    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt"]
Sep 16 19:23:54.623815 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.623791    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1323e73d-f926-45ef-9891-b8c6096d673e-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt\" (UID: \"1323e73d-f926-45ef-9891-b8c6096d673e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt"
Sep 16 19:23:54.623960 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.623845    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc64f\" (UniqueName: \"kubernetes.io/projected/1323e73d-f926-45ef-9891-b8c6096d673e-kube-api-access-xc64f\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt\" (UID: \"1323e73d-f926-45ef-9891-b8c6096d673e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt"
Sep 16 19:23:54.624009 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.623957    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1323e73d-f926-45ef-9891-b8c6096d673e-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt\" (UID: \"1323e73d-f926-45ef-9891-b8c6096d673e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt"
Sep 16 19:23:54.624009 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.623986    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1323e73d-f926-45ef-9891-b8c6096d673e-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt\" (UID: \"1323e73d-f926-45ef-9891-b8c6096d673e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt"
Sep 16 19:23:54.624077 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.624006    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1323e73d-f926-45ef-9891-b8c6096d673e-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt\" (UID: \"1323e73d-f926-45ef-9891-b8c6096d673e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt"
Sep 16 19:23:54.724980 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.724924    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1323e73d-f926-45ef-9891-b8c6096d673e-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt\" (UID: \"1323e73d-f926-45ef-9891-b8c6096d673e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt"
Sep 16 19:23:54.724980 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.724968    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1323e73d-f926-45ef-9891-b8c6096d673e-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt\" (UID: \"1323e73d-f926-45ef-9891-b8c6096d673e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt"
Sep 16 19:23:54.725118 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.724992    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1323e73d-f926-45ef-9891-b8c6096d673e-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt\" (UID: \"1323e73d-f926-45ef-9891-b8c6096d673e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt"
Sep 16 19:23:54.725118 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.725037    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1323e73d-f926-45ef-9891-b8c6096d673e-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt\" (UID: \"1323e73d-f926-45ef-9891-b8c6096d673e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt"
Sep 16 19:23:54.725118 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.725093    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc64f\" (UniqueName: \"kubernetes.io/projected/1323e73d-f926-45ef-9891-b8c6096d673e-kube-api-access-xc64f\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt\" (UID: \"1323e73d-f926-45ef-9891-b8c6096d673e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt"
Sep 16 19:23:54.725461 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.725440    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1323e73d-f926-45ef-9891-b8c6096d673e-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt\" (UID: \"1323e73d-f926-45ef-9891-b8c6096d673e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt"
Sep 16 19:23:54.725658 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.725636    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1323e73d-f926-45ef-9891-b8c6096d673e-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt\" (UID: \"1323e73d-f926-45ef-9891-b8c6096d673e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt"
Sep 16 19:23:54.725658 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.725648    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1323e73d-f926-45ef-9891-b8c6096d673e-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt\" (UID: \"1323e73d-f926-45ef-9891-b8c6096d673e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt"
Sep 16 19:23:54.727317 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.727300    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1323e73d-f926-45ef-9891-b8c6096d673e-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt\" (UID: \"1323e73d-f926-45ef-9891-b8c6096d673e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt"
Sep 16 19:23:54.733475 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.733453    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc64f\" (UniqueName: \"kubernetes.io/projected/1323e73d-f926-45ef-9891-b8c6096d673e-kube-api-access-xc64f\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt\" (UID: \"1323e73d-f926-45ef-9891-b8c6096d673e\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt"
Sep 16 19:23:54.776207 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.776179    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt"
Sep 16 19:23:54.876406 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.876384    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98"
Sep 16 19:23:54.913344 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.913315    2554 generic.go:356] "Generic (PLEG): container finished" podID="0bf1ae1a-b155-4603-af80-eb136de46752" containerID="ec9f338f4449847357993cdbea69b213b02288bf2ed6ab36af9d1e680bff3e72" exitCode=0
Sep 16 19:23:54.913496 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.913362    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98" event={"ID":"0bf1ae1a-b155-4603-af80-eb136de46752","Type":"ContainerDied","Data":"ec9f338f4449847357993cdbea69b213b02288bf2ed6ab36af9d1e680bff3e72"}
Sep 16 19:23:54.913496 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.913384    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98"
Sep 16 19:23:54.913496 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.913429    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98" event={"ID":"0bf1ae1a-b155-4603-af80-eb136de46752","Type":"ContainerDied","Data":"8f4130075abb66798f4dbeb47911035adc2d068aeb9f259d2fbe211f1999e13c"}
Sep 16 19:23:54.913496 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.913455    2554 scope.go:117] "RemoveContainer" containerID="2d5db325fd1c631a958ae1e4a0448b54f65a80b0d2d810b24fa3c67bdc7e6697"
Sep 16 19:23:54.923768 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.923750    2554 scope.go:117] "RemoveContainer" containerID="ec9f338f4449847357993cdbea69b213b02288bf2ed6ab36af9d1e680bff3e72"
Sep 16 19:23:54.931156 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.931139    2554 scope.go:117] "RemoveContainer" containerID="39b5d63cdf8925615ad4c95a82021efe0e5059359f420d78635a7cc181f4b9eb"
Sep 16 19:23:54.938175 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.938160    2554 scope.go:117] "RemoveContainer" containerID="2d5db325fd1c631a958ae1e4a0448b54f65a80b0d2d810b24fa3c67bdc7e6697"
Sep 16 19:23:54.938407 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:23:54.938389    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d5db325fd1c631a958ae1e4a0448b54f65a80b0d2d810b24fa3c67bdc7e6697\": container with ID starting with 2d5db325fd1c631a958ae1e4a0448b54f65a80b0d2d810b24fa3c67bdc7e6697 not found: ID does not exist" containerID="2d5db325fd1c631a958ae1e4a0448b54f65a80b0d2d810b24fa3c67bdc7e6697"
Sep 16 19:23:54.938474 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.938413    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d5db325fd1c631a958ae1e4a0448b54f65a80b0d2d810b24fa3c67bdc7e6697"} err="failed to get container status \"2d5db325fd1c631a958ae1e4a0448b54f65a80b0d2d810b24fa3c67bdc7e6697\": rpc error: code = NotFound desc = could not find container \"2d5db325fd1c631a958ae1e4a0448b54f65a80b0d2d810b24fa3c67bdc7e6697\": container with ID starting with 2d5db325fd1c631a958ae1e4a0448b54f65a80b0d2d810b24fa3c67bdc7e6697 not found: ID does not exist"
Sep 16 19:23:54.938474 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.938428    2554 scope.go:117] "RemoveContainer" containerID="ec9f338f4449847357993cdbea69b213b02288bf2ed6ab36af9d1e680bff3e72"
Sep 16 19:23:54.938672 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:23:54.938655    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec9f338f4449847357993cdbea69b213b02288bf2ed6ab36af9d1e680bff3e72\": container with ID starting with ec9f338f4449847357993cdbea69b213b02288bf2ed6ab36af9d1e680bff3e72 not found: ID does not exist" containerID="ec9f338f4449847357993cdbea69b213b02288bf2ed6ab36af9d1e680bff3e72"
Sep 16 19:23:54.938712 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.938677    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec9f338f4449847357993cdbea69b213b02288bf2ed6ab36af9d1e680bff3e72"} err="failed to get container status \"ec9f338f4449847357993cdbea69b213b02288bf2ed6ab36af9d1e680bff3e72\": rpc error: code = NotFound desc = could not find container \"ec9f338f4449847357993cdbea69b213b02288bf2ed6ab36af9d1e680bff3e72\": container with ID starting with ec9f338f4449847357993cdbea69b213b02288bf2ed6ab36af9d1e680bff3e72 not found: ID does not exist"
Sep 16 19:23:54.938712 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.938693    2554 scope.go:117] "RemoveContainer" containerID="39b5d63cdf8925615ad4c95a82021efe0e5059359f420d78635a7cc181f4b9eb"
Sep 16 19:23:54.938966 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:23:54.938948    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39b5d63cdf8925615ad4c95a82021efe0e5059359f420d78635a7cc181f4b9eb\": container with ID starting with 39b5d63cdf8925615ad4c95a82021efe0e5059359f420d78635a7cc181f4b9eb not found: ID does not exist" containerID="39b5d63cdf8925615ad4c95a82021efe0e5059359f420d78635a7cc181f4b9eb"
Sep 16 19:23:54.939006 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:54.938973    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39b5d63cdf8925615ad4c95a82021efe0e5059359f420d78635a7cc181f4b9eb"} err="failed to get container status \"39b5d63cdf8925615ad4c95a82021efe0e5059359f420d78635a7cc181f4b9eb\": rpc error: code = NotFound desc = could not find container \"39b5d63cdf8925615ad4c95a82021efe0e5059359f420d78635a7cc181f4b9eb\": container with ID starting with 39b5d63cdf8925615ad4c95a82021efe0e5059359f420d78635a7cc181f4b9eb not found: ID does not exist"
Sep 16 19:23:55.028991 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:55.028968    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6kzj\" (UniqueName: \"kubernetes.io/projected/0bf1ae1a-b155-4603-af80-eb136de46752-kube-api-access-b6kzj\") pod \"0bf1ae1a-b155-4603-af80-eb136de46752\" (UID: \"0bf1ae1a-b155-4603-af80-eb136de46752\") "
Sep 16 19:23:55.029079 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:55.029014    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bf1ae1a-b155-4603-af80-eb136de46752-proxy-tls\") pod \"0bf1ae1a-b155-4603-af80-eb136de46752\" (UID: \"0bf1ae1a-b155-4603-af80-eb136de46752\") "
Sep 16 19:23:55.029079 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:55.029050    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bf1ae1a-b155-4603-af80-eb136de46752-kube-rbac-proxy-sar-config\") pod \"0bf1ae1a-b155-4603-af80-eb136de46752\" (UID: \"0bf1ae1a-b155-4603-af80-eb136de46752\") "
Sep 16 19:23:55.029079 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:55.029078    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bf1ae1a-b155-4603-af80-eb136de46752-kserve-provision-location\") pod \"0bf1ae1a-b155-4603-af80-eb136de46752\" (UID: \"0bf1ae1a-b155-4603-af80-eb136de46752\") "
Sep 16 19:23:55.029194 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:55.029107    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bf1ae1a-b155-4603-af80-eb136de46752-cabundle-cert\") pod \"0bf1ae1a-b155-4603-af80-eb136de46752\" (UID: \"0bf1ae1a-b155-4603-af80-eb136de46752\") "
Sep 16 19:23:55.029440 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:55.029413    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bf1ae1a-b155-4603-af80-eb136de46752-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0bf1ae1a-b155-4603-af80-eb136de46752" (UID: "0bf1ae1a-b155-4603-af80-eb136de46752"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:23:55.029440 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:55.029425    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bf1ae1a-b155-4603-af80-eb136de46752-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0bf1ae1a-b155-4603-af80-eb136de46752" (UID: "0bf1ae1a-b155-4603-af80-eb136de46752"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:23:55.029580 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:55.029490    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bf1ae1a-b155-4603-af80-eb136de46752-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0bf1ae1a-b155-4603-af80-eb136de46752" (UID: "0bf1ae1a-b155-4603-af80-eb136de46752"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:23:55.031168 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:55.031146    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bf1ae1a-b155-4603-af80-eb136de46752-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0bf1ae1a-b155-4603-af80-eb136de46752" (UID: "0bf1ae1a-b155-4603-af80-eb136de46752"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:23:55.031283 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:55.031164    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bf1ae1a-b155-4603-af80-eb136de46752-kube-api-access-b6kzj" (OuterVolumeSpecName: "kube-api-access-b6kzj") pod "0bf1ae1a-b155-4603-af80-eb136de46752" (UID: "0bf1ae1a-b155-4603-af80-eb136de46752"). InnerVolumeSpecName "kube-api-access-b6kzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:23:55.055349 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:55.055329    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt"]
Sep 16 19:23:55.057138 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:23:55.057118    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1323e73d_f926_45ef_9891_b8c6096d673e.slice/crio-1ae2c4ad334fc3133f29cca0ca24d45957c7f39f4ffb7832602544cad5931541 WatchSource:0}: Error finding container 1ae2c4ad334fc3133f29cca0ca24d45957c7f39f4ffb7832602544cad5931541: Status 404 returned error can't find the container with id 1ae2c4ad334fc3133f29cca0ca24d45957c7f39f4ffb7832602544cad5931541
Sep 16 19:23:55.130288 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:55.130265    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6kzj\" (UniqueName: \"kubernetes.io/projected/0bf1ae1a-b155-4603-af80-eb136de46752-kube-api-access-b6kzj\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:23:55.130288 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:55.130287    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0bf1ae1a-b155-4603-af80-eb136de46752-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:23:55.130410 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:55.130298    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0bf1ae1a-b155-4603-af80-eb136de46752-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:23:55.130410 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:55.130307    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0bf1ae1a-b155-4603-af80-eb136de46752-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:23:55.130410 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:55.130316    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0bf1ae1a-b155-4603-af80-eb136de46752-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:23:55.234756 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:55.234731    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98"]
Sep 16 19:23:55.241134 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:55.241113    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-m9q98"]
Sep 16 19:23:55.920487 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:55.920453    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt" event={"ID":"1323e73d-f926-45ef-9891-b8c6096d673e","Type":"ContainerStarted","Data":"a7e4c1b4afda7ca54b254500d7212d034811d83c761fa976ee3e773daa3c2110"}
Sep 16 19:23:55.920885 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:55.920515    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt" event={"ID":"1323e73d-f926-45ef-9891-b8c6096d673e","Type":"ContainerStarted","Data":"1ae2c4ad334fc3133f29cca0ca24d45957c7f39f4ffb7832602544cad5931541"}
Sep 16 19:23:56.443765 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:56.443711    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bf1ae1a-b155-4603-af80-eb136de46752" path="/var/lib/kubelet/pods/0bf1ae1a-b155-4603-af80-eb136de46752/volumes"
Sep 16 19:23:58.612330 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:58.612295    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf"]
Sep 16 19:23:58.612745 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:58.612593    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf" podUID="417f90db-905b-45bd-8cb6-80bb7c7410c8" containerName="kserve-container" containerID="cri-o://10750c19050a6211b6c17d8cd066e29f335e2fa58d8263d5b7e0fae1280ceceb" gracePeriod=30
Sep 16 19:23:58.612830 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:58.612646    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf" podUID="417f90db-905b-45bd-8cb6-80bb7c7410c8" containerName="kube-rbac-proxy" containerID="cri-o://c7b77f19e4d17a8ef17ae0ff4b154cfaac0fa717c013669e432517189e3d740d" gracePeriod=30
Sep 16 19:23:58.613552 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:58.613529    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf"
Sep 16 19:23:58.937976 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:58.937903    2554 generic.go:356] "Generic (PLEG): container finished" podID="417f90db-905b-45bd-8cb6-80bb7c7410c8" containerID="c7b77f19e4d17a8ef17ae0ff4b154cfaac0fa717c013669e432517189e3d740d" exitCode=2
Sep 16 19:23:58.938103 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:58.937980    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf" event={"ID":"417f90db-905b-45bd-8cb6-80bb7c7410c8","Type":"ContainerDied","Data":"c7b77f19e4d17a8ef17ae0ff4b154cfaac0fa717c013669e432517189e3d740d"}
Sep 16 19:23:59.555936 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:59.555895    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf" podUID="417f90db-905b-45bd-8cb6-80bb7c7410c8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Sep 16 19:23:59.944561 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:59.944481    2554 generic.go:356] "Generic (PLEG): container finished" podID="1323e73d-f926-45ef-9891-b8c6096d673e" containerID="a7e4c1b4afda7ca54b254500d7212d034811d83c761fa976ee3e773daa3c2110" exitCode=0
Sep 16 19:23:59.944933 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:23:59.944556    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt" event={"ID":"1323e73d-f926-45ef-9891-b8c6096d673e","Type":"ContainerDied","Data":"a7e4c1b4afda7ca54b254500d7212d034811d83c761fa976ee3e773daa3c2110"}
Sep 16 19:24:00.952957 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:00.952920    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt" event={"ID":"1323e73d-f926-45ef-9891-b8c6096d673e","Type":"ContainerStarted","Data":"c74bde6e6d5128730d85f19da0e935670e1e467398159940cbe7cd79ced8d62b"}
Sep 16 19:24:00.953355 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:00.952983    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt" event={"ID":"1323e73d-f926-45ef-9891-b8c6096d673e","Type":"ContainerStarted","Data":"18278d195c3dd6756d36221c68623f4a8709724164a1f021011053a1633cd79a"}
Sep 16 19:24:00.953355 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:00.953191    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt"
Sep 16 19:24:00.953355 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:00.953241    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt"
Sep 16 19:24:00.995825 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:00.995780    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt" podStartSLOduration=3.064913273 podStartE2EDuration="6.995766694s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.930853421s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:24:00.995575115 +0000 UTC m=+2351.071344799" watchObservedRunningTime="2026-09-16 19:24:00.995766694 +0000 UTC m=+2351.071536325"
Sep 16 19:24:02.646528 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.646507    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf"
Sep 16 19:24:02.786257 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.786233    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/417f90db-905b-45bd-8cb6-80bb7c7410c8-cabundle-cert\") pod \"417f90db-905b-45bd-8cb6-80bb7c7410c8\" (UID: \"417f90db-905b-45bd-8cb6-80bb7c7410c8\") "
Sep 16 19:24:02.786408 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.786288    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/417f90db-905b-45bd-8cb6-80bb7c7410c8-kube-rbac-proxy-sar-config\") pod \"417f90db-905b-45bd-8cb6-80bb7c7410c8\" (UID: \"417f90db-905b-45bd-8cb6-80bb7c7410c8\") "
Sep 16 19:24:02.786408 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.786318    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/417f90db-905b-45bd-8cb6-80bb7c7410c8-kserve-provision-location\") pod \"417f90db-905b-45bd-8cb6-80bb7c7410c8\" (UID: \"417f90db-905b-45bd-8cb6-80bb7c7410c8\") "
Sep 16 19:24:02.786408 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.786340    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/417f90db-905b-45bd-8cb6-80bb7c7410c8-proxy-tls\") pod \"417f90db-905b-45bd-8cb6-80bb7c7410c8\" (UID: \"417f90db-905b-45bd-8cb6-80bb7c7410c8\") "
Sep 16 19:24:02.786408 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.786361    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddlkk\" (UniqueName: \"kubernetes.io/projected/417f90db-905b-45bd-8cb6-80bb7c7410c8-kube-api-access-ddlkk\") pod \"417f90db-905b-45bd-8cb6-80bb7c7410c8\" (UID: \"417f90db-905b-45bd-8cb6-80bb7c7410c8\") "
Sep 16 19:24:02.786642 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.786618    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/417f90db-905b-45bd-8cb6-80bb7c7410c8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "417f90db-905b-45bd-8cb6-80bb7c7410c8" (UID: "417f90db-905b-45bd-8cb6-80bb7c7410c8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:24:02.786700 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.786662    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/417f90db-905b-45bd-8cb6-80bb7c7410c8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "417f90db-905b-45bd-8cb6-80bb7c7410c8" (UID: "417f90db-905b-45bd-8cb6-80bb7c7410c8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:02.786771 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.786699    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/417f90db-905b-45bd-8cb6-80bb7c7410c8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "417f90db-905b-45bd-8cb6-80bb7c7410c8" (UID: "417f90db-905b-45bd-8cb6-80bb7c7410c8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:24:02.788446 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.788420    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/417f90db-905b-45bd-8cb6-80bb7c7410c8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "417f90db-905b-45bd-8cb6-80bb7c7410c8" (UID: "417f90db-905b-45bd-8cb6-80bb7c7410c8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:24:02.788614 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.788594    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/417f90db-905b-45bd-8cb6-80bb7c7410c8-kube-api-access-ddlkk" (OuterVolumeSpecName: "kube-api-access-ddlkk") pod "417f90db-905b-45bd-8cb6-80bb7c7410c8" (UID: "417f90db-905b-45bd-8cb6-80bb7c7410c8"). InnerVolumeSpecName "kube-api-access-ddlkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:24:02.887100 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.887074    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/417f90db-905b-45bd-8cb6-80bb7c7410c8-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:24:02.887100 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.887101    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/417f90db-905b-45bd-8cb6-80bb7c7410c8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:24:02.887219 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.887111    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/417f90db-905b-45bd-8cb6-80bb7c7410c8-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:24:02.887219 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.887120    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/417f90db-905b-45bd-8cb6-80bb7c7410c8-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:24:02.887219 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.887131    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddlkk\" (UniqueName: \"kubernetes.io/projected/417f90db-905b-45bd-8cb6-80bb7c7410c8-kube-api-access-ddlkk\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:24:02.965278 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.965248    2554 generic.go:356] "Generic (PLEG): container finished" podID="417f90db-905b-45bd-8cb6-80bb7c7410c8" containerID="10750c19050a6211b6c17d8cd066e29f335e2fa58d8263d5b7e0fae1280ceceb" exitCode=0
Sep 16 19:24:02.965388 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.965292    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf" event={"ID":"417f90db-905b-45bd-8cb6-80bb7c7410c8","Type":"ContainerDied","Data":"10750c19050a6211b6c17d8cd066e29f335e2fa58d8263d5b7e0fae1280ceceb"}
Sep 16 19:24:02.965388 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.965316    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf"
Sep 16 19:24:02.965388 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.965361    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf" event={"ID":"417f90db-905b-45bd-8cb6-80bb7c7410c8","Type":"ContainerDied","Data":"44639d92b39d2a897c56adbfb94e040a9cbd49f7761bd6ca891046cc0cb1695c"}
Sep 16 19:24:02.965551 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.965391    2554 scope.go:117] "RemoveContainer" containerID="c7b77f19e4d17a8ef17ae0ff4b154cfaac0fa717c013669e432517189e3d740d"
Sep 16 19:24:02.974764 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.974748    2554 scope.go:117] "RemoveContainer" containerID="10750c19050a6211b6c17d8cd066e29f335e2fa58d8263d5b7e0fae1280ceceb"
Sep 16 19:24:02.982090 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.982074    2554 scope.go:117] "RemoveContainer" containerID="756c845dc6d5cd7b564d532a9c2fc3664cea4d54184a0d4be9b54d324d53bec6"
Sep 16 19:24:02.987646 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.987622    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf"]
Sep 16 19:24:02.990021 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.990002    2554 scope.go:117] "RemoveContainer" containerID="c7b77f19e4d17a8ef17ae0ff4b154cfaac0fa717c013669e432517189e3d740d"
Sep 16 19:24:02.990299 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:24:02.990279    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7b77f19e4d17a8ef17ae0ff4b154cfaac0fa717c013669e432517189e3d740d\": container with ID starting with c7b77f19e4d17a8ef17ae0ff4b154cfaac0fa717c013669e432517189e3d740d not found: ID does not exist" containerID="c7b77f19e4d17a8ef17ae0ff4b154cfaac0fa717c013669e432517189e3d740d"
Sep 16 19:24:02.990379 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.990309    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7b77f19e4d17a8ef17ae0ff4b154cfaac0fa717c013669e432517189e3d740d"} err="failed to get container status \"c7b77f19e4d17a8ef17ae0ff4b154cfaac0fa717c013669e432517189e3d740d\": rpc error: code = NotFound desc = could not find container \"c7b77f19e4d17a8ef17ae0ff4b154cfaac0fa717c013669e432517189e3d740d\": container with ID starting with c7b77f19e4d17a8ef17ae0ff4b154cfaac0fa717c013669e432517189e3d740d not found: ID does not exist"
Sep 16 19:24:02.990379 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.990325    2554 scope.go:117] "RemoveContainer" containerID="10750c19050a6211b6c17d8cd066e29f335e2fa58d8263d5b7e0fae1280ceceb"
Sep 16 19:24:02.990592 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:24:02.990574    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10750c19050a6211b6c17d8cd066e29f335e2fa58d8263d5b7e0fae1280ceceb\": container with ID starting with 10750c19050a6211b6c17d8cd066e29f335e2fa58d8263d5b7e0fae1280ceceb not found: ID does not exist" containerID="10750c19050a6211b6c17d8cd066e29f335e2fa58d8263d5b7e0fae1280ceceb"
Sep 16 19:24:02.990644 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.990597    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10750c19050a6211b6c17d8cd066e29f335e2fa58d8263d5b7e0fae1280ceceb"} err="failed to get container status \"10750c19050a6211b6c17d8cd066e29f335e2fa58d8263d5b7e0fae1280ceceb\": rpc error: code = NotFound desc = could not find container \"10750c19050a6211b6c17d8cd066e29f335e2fa58d8263d5b7e0fae1280ceceb\": container with ID starting with 10750c19050a6211b6c17d8cd066e29f335e2fa58d8263d5b7e0fae1280ceceb not found: ID does not exist"
Sep 16 19:24:02.990644 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.990611    2554 scope.go:117] "RemoveContainer" containerID="756c845dc6d5cd7b564d532a9c2fc3664cea4d54184a0d4be9b54d324d53bec6"
Sep 16 19:24:02.990905 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:24:02.990888    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"756c845dc6d5cd7b564d532a9c2fc3664cea4d54184a0d4be9b54d324d53bec6\": container with ID starting with 756c845dc6d5cd7b564d532a9c2fc3664cea4d54184a0d4be9b54d324d53bec6 not found: ID does not exist" containerID="756c845dc6d5cd7b564d532a9c2fc3664cea4d54184a0d4be9b54d324d53bec6"
Sep 16 19:24:02.990970 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.990908    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"756c845dc6d5cd7b564d532a9c2fc3664cea4d54184a0d4be9b54d324d53bec6"} err="failed to get container status \"756c845dc6d5cd7b564d532a9c2fc3664cea4d54184a0d4be9b54d324d53bec6\": rpc error: code = NotFound desc = could not find container \"756c845dc6d5cd7b564d532a9c2fc3664cea4d54184a0d4be9b54d324d53bec6\": container with ID starting with 756c845dc6d5cd7b564d532a9c2fc3664cea4d54184a0d4be9b54d324d53bec6 not found: ID does not exist"
Sep 16 19:24:02.991177 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:02.991162    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57d5c586ff-2hbzf"]
Sep 16 19:24:04.443565 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:04.443528    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="417f90db-905b-45bd-8cb6-80bb7c7410c8" path="/var/lib/kubelet/pods/417f90db-905b-45bd-8cb6-80bb7c7410c8/volumes"
Sep 16 19:24:06.962889 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:06.962862    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt"
Sep 16 19:24:36.966992 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:36.966964    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt"
Sep 16 19:24:44.521684 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:44.521653    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt"]
Sep 16 19:24:44.522122 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:44.521910    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt" podUID="1323e73d-f926-45ef-9891-b8c6096d673e" containerName="kserve-container" containerID="cri-o://18278d195c3dd6756d36221c68623f4a8709724164a1f021011053a1633cd79a" gracePeriod=30
Sep 16 19:24:44.522122 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:44.521979    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt" podUID="1323e73d-f926-45ef-9891-b8c6096d673e" containerName="kube-rbac-proxy" containerID="cri-o://c74bde6e6d5128730d85f19da0e935670e1e467398159940cbe7cd79ced8d62b" gracePeriod=30
Sep 16 19:24:45.163471 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:45.163439    2554 generic.go:356] "Generic (PLEG): container finished" podID="1323e73d-f926-45ef-9891-b8c6096d673e" containerID="c74bde6e6d5128730d85f19da0e935670e1e467398159940cbe7cd79ced8d62b" exitCode=2
Sep 16 19:24:45.163658 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:45.163518    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt" event={"ID":"1323e73d-f926-45ef-9891-b8c6096d673e","Type":"ContainerDied","Data":"c74bde6e6d5128730d85f19da0e935670e1e467398159940cbe7cd79ced8d62b"}
Sep 16 19:24:46.958709 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:46.958664    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt" podUID="1323e73d-f926-45ef-9891-b8c6096d673e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
Sep 16 19:24:46.964104 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:46.964078    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt" podUID="1323e73d-f926-45ef-9891-b8c6096d673e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.56:8080/v2/models/xgboost-v2-mlserver/ready\": dial tcp 10.134.0.56:8080: connect: connection refused"
Sep 16 19:24:50.302894 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:24:50.302866    2554 info.go:109] Failed to get network devices: open /sys/class/net/1ae2c4ad334fc31/address: no such file or directory
Sep 16 19:24:50.358820 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:50.358798    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt"
Sep 16 19:24:50.454814 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:50.454737    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1323e73d-f926-45ef-9891-b8c6096d673e-kube-rbac-proxy-sar-config\") pod \"1323e73d-f926-45ef-9891-b8c6096d673e\" (UID: \"1323e73d-f926-45ef-9891-b8c6096d673e\") "
Sep 16 19:24:50.454814 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:50.454791    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1323e73d-f926-45ef-9891-b8c6096d673e-kserve-provision-location\") pod \"1323e73d-f926-45ef-9891-b8c6096d673e\" (UID: \"1323e73d-f926-45ef-9891-b8c6096d673e\") "
Sep 16 19:24:50.455026 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:50.454833    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1323e73d-f926-45ef-9891-b8c6096d673e-cabundle-cert\") pod \"1323e73d-f926-45ef-9891-b8c6096d673e\" (UID: \"1323e73d-f926-45ef-9891-b8c6096d673e\") "
Sep 16 19:24:50.455026 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:50.454852    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1323e73d-f926-45ef-9891-b8c6096d673e-proxy-tls\") pod \"1323e73d-f926-45ef-9891-b8c6096d673e\" (UID: \"1323e73d-f926-45ef-9891-b8c6096d673e\") "
Sep 16 19:24:50.455026 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:50.454887    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xc64f\" (UniqueName: \"kubernetes.io/projected/1323e73d-f926-45ef-9891-b8c6096d673e-kube-api-access-xc64f\") pod \"1323e73d-f926-45ef-9891-b8c6096d673e\" (UID: \"1323e73d-f926-45ef-9891-b8c6096d673e\") "
Sep 16 19:24:50.455026 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:50.455012    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1323e73d-f926-45ef-9891-b8c6096d673e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1323e73d-f926-45ef-9891-b8c6096d673e" (UID: "1323e73d-f926-45ef-9891-b8c6096d673e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:24:50.455250 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:50.455100    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1323e73d-f926-45ef-9891-b8c6096d673e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1323e73d-f926-45ef-9891-b8c6096d673e" (UID: "1323e73d-f926-45ef-9891-b8c6096d673e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:24:50.455250 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:50.455212    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1323e73d-f926-45ef-9891-b8c6096d673e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:24:50.455250 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:50.455226    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1323e73d-f926-45ef-9891-b8c6096d673e-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:24:50.455250 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:50.455236    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1323e73d-f926-45ef-9891-b8c6096d673e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1323e73d-f926-45ef-9891-b8c6096d673e" (UID: "1323e73d-f926-45ef-9891-b8c6096d673e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:24:50.456767 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:50.456748    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1323e73d-f926-45ef-9891-b8c6096d673e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1323e73d-f926-45ef-9891-b8c6096d673e" (UID: "1323e73d-f926-45ef-9891-b8c6096d673e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:24:50.456893 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:50.456876    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1323e73d-f926-45ef-9891-b8c6096d673e-kube-api-access-xc64f" (OuterVolumeSpecName: "kube-api-access-xc64f") pod "1323e73d-f926-45ef-9891-b8c6096d673e" (UID: "1323e73d-f926-45ef-9891-b8c6096d673e"). InnerVolumeSpecName "kube-api-access-xc64f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:24:50.552609 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:50.552584    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/2.log"
Sep 16 19:24:50.555695 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:50.555670    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc64f\" (UniqueName: \"kubernetes.io/projected/1323e73d-f926-45ef-9891-b8c6096d673e-kube-api-access-xc64f\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:24:50.555828 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:50.555699    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1323e73d-f926-45ef-9891-b8c6096d673e-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:24:50.555828 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:50.555714    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1323e73d-f926-45ef-9891-b8c6096d673e-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:24:50.557482 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:50.557463    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/2.log"
Sep 16 19:24:50.558538 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:50.558521    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovn-acl-logging/0.log"
Sep 16 19:24:50.562881 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:50.562863    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovn-acl-logging/0.log"
Sep 16 19:24:51.191464 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:51.191430    2554 generic.go:356] "Generic (PLEG): container finished" podID="1323e73d-f926-45ef-9891-b8c6096d673e" containerID="18278d195c3dd6756d36221c68623f4a8709724164a1f021011053a1633cd79a" exitCode=0
Sep 16 19:24:51.191647 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:51.191486    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt"
Sep 16 19:24:51.191647 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:51.191511    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt" event={"ID":"1323e73d-f926-45ef-9891-b8c6096d673e","Type":"ContainerDied","Data":"18278d195c3dd6756d36221c68623f4a8709724164a1f021011053a1633cd79a"}
Sep 16 19:24:51.191647 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:51.191567    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt" event={"ID":"1323e73d-f926-45ef-9891-b8c6096d673e","Type":"ContainerDied","Data":"1ae2c4ad334fc3133f29cca0ca24d45957c7f39f4ffb7832602544cad5931541"}
Sep 16 19:24:51.191647 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:51.191594    2554 scope.go:117] "RemoveContainer" containerID="c74bde6e6d5128730d85f19da0e935670e1e467398159940cbe7cd79ced8d62b"
Sep 16 19:24:51.201449 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:51.201431    2554 scope.go:117] "RemoveContainer" containerID="18278d195c3dd6756d36221c68623f4a8709724164a1f021011053a1633cd79a"
Sep 16 19:24:51.208709 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:51.208691    2554 scope.go:117] "RemoveContainer" containerID="a7e4c1b4afda7ca54b254500d7212d034811d83c761fa976ee3e773daa3c2110"
Sep 16 19:24:51.215306 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:51.215281    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt"]
Sep 16 19:24:51.216995 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:51.216976    2554 scope.go:117] "RemoveContainer" containerID="c74bde6e6d5128730d85f19da0e935670e1e467398159940cbe7cd79ced8d62b"
Sep 16 19:24:51.217261 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:24:51.217240    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c74bde6e6d5128730d85f19da0e935670e1e467398159940cbe7cd79ced8d62b\": container with ID starting with c74bde6e6d5128730d85f19da0e935670e1e467398159940cbe7cd79ced8d62b not found: ID does not exist" containerID="c74bde6e6d5128730d85f19da0e935670e1e467398159940cbe7cd79ced8d62b"
Sep 16 19:24:51.217364 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:51.217267    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c74bde6e6d5128730d85f19da0e935670e1e467398159940cbe7cd79ced8d62b"} err="failed to get container status \"c74bde6e6d5128730d85f19da0e935670e1e467398159940cbe7cd79ced8d62b\": rpc error: code = NotFound desc = could not find container \"c74bde6e6d5128730d85f19da0e935670e1e467398159940cbe7cd79ced8d62b\": container with ID starting with c74bde6e6d5128730d85f19da0e935670e1e467398159940cbe7cd79ced8d62b not found: ID does not exist"
Sep 16 19:24:51.217364 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:51.217284    2554 scope.go:117] "RemoveContainer" containerID="18278d195c3dd6756d36221c68623f4a8709724164a1f021011053a1633cd79a"
Sep 16 19:24:51.217555 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:24:51.217533    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18278d195c3dd6756d36221c68623f4a8709724164a1f021011053a1633cd79a\": container with ID starting with 18278d195c3dd6756d36221c68623f4a8709724164a1f021011053a1633cd79a not found: ID does not exist" containerID="18278d195c3dd6756d36221c68623f4a8709724164a1f021011053a1633cd79a"
Sep 16 19:24:51.217606 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:51.217565    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18278d195c3dd6756d36221c68623f4a8709724164a1f021011053a1633cd79a"} err="failed to get container status \"18278d195c3dd6756d36221c68623f4a8709724164a1f021011053a1633cd79a\": rpc error: code = NotFound desc = could not find container \"18278d195c3dd6756d36221c68623f4a8709724164a1f021011053a1633cd79a\": container with ID starting with 18278d195c3dd6756d36221c68623f4a8709724164a1f021011053a1633cd79a not found: ID does not exist"
Sep 16 19:24:51.217606 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:51.217586    2554 scope.go:117] "RemoveContainer" containerID="a7e4c1b4afda7ca54b254500d7212d034811d83c761fa976ee3e773daa3c2110"
Sep 16 19:24:51.217873 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:24:51.217854    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7e4c1b4afda7ca54b254500d7212d034811d83c761fa976ee3e773daa3c2110\": container with ID starting with a7e4c1b4afda7ca54b254500d7212d034811d83c761fa976ee3e773daa3c2110 not found: ID does not exist" containerID="a7e4c1b4afda7ca54b254500d7212d034811d83c761fa976ee3e773daa3c2110"
Sep 16 19:24:51.217962 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:51.217876    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7e4c1b4afda7ca54b254500d7212d034811d83c761fa976ee3e773daa3c2110"} err="failed to get container status \"a7e4c1b4afda7ca54b254500d7212d034811d83c761fa976ee3e773daa3c2110\": rpc error: code = NotFound desc = could not find container \"a7e4c1b4afda7ca54b254500d7212d034811d83c761fa976ee3e773daa3c2110\": container with ID starting with a7e4c1b4afda7ca54b254500d7212d034811d83c761fa976ee3e773daa3c2110 not found: ID does not exist"
Sep 16 19:24:51.218256 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:51.218241    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-xdhkt"]
Sep 16 19:24:52.443100 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:24:52.443063    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1323e73d-f926-45ef-9891-b8c6096d673e" path="/var/lib/kubelet/pods/1323e73d-f926-45ef-9891-b8c6096d673e/volumes"
Sep 16 19:25:51.534822 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.534786    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq"]
Sep 16 19:25:51.535288 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.535116    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="417f90db-905b-45bd-8cb6-80bb7c7410c8" containerName="kube-rbac-proxy"
Sep 16 19:25:51.535288 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.535128    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="417f90db-905b-45bd-8cb6-80bb7c7410c8" containerName="kube-rbac-proxy"
Sep 16 19:25:51.535288 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.535142    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1323e73d-f926-45ef-9891-b8c6096d673e" containerName="kube-rbac-proxy"
Sep 16 19:25:51.535288 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.535147    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1323e73d-f926-45ef-9891-b8c6096d673e" containerName="kube-rbac-proxy"
Sep 16 19:25:51.535288 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.535156    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bf1ae1a-b155-4603-af80-eb136de46752" containerName="kserve-container"
Sep 16 19:25:51.535288 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.535162    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bf1ae1a-b155-4603-af80-eb136de46752" containerName="kserve-container"
Sep 16 19:25:51.535288 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.535168    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="417f90db-905b-45bd-8cb6-80bb7c7410c8" containerName="storage-initializer"
Sep 16 19:25:51.535288 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.535175    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="417f90db-905b-45bd-8cb6-80bb7c7410c8" containerName="storage-initializer"
Sep 16 19:25:51.535288 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.535184    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="417f90db-905b-45bd-8cb6-80bb7c7410c8" containerName="kserve-container"
Sep 16 19:25:51.535288 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.535189    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="417f90db-905b-45bd-8cb6-80bb7c7410c8" containerName="kserve-container"
Sep 16 19:25:51.535288 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.535194    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1323e73d-f926-45ef-9891-b8c6096d673e" containerName="storage-initializer"
Sep 16 19:25:51.535288 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.535200    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1323e73d-f926-45ef-9891-b8c6096d673e" containerName="storage-initializer"
Sep 16 19:25:51.535288 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.535210    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bf1ae1a-b155-4603-af80-eb136de46752" containerName="kube-rbac-proxy"
Sep 16 19:25:51.535288 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.535215    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bf1ae1a-b155-4603-af80-eb136de46752" containerName="kube-rbac-proxy"
Sep 16 19:25:51.535288 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.535225    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bf1ae1a-b155-4603-af80-eb136de46752" containerName="storage-initializer"
Sep 16 19:25:51.535288 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.535232    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bf1ae1a-b155-4603-af80-eb136de46752" containerName="storage-initializer"
Sep 16 19:25:51.535288 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.535237    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1323e73d-f926-45ef-9891-b8c6096d673e" containerName="kserve-container"
Sep 16 19:25:51.535288 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.535242    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1323e73d-f926-45ef-9891-b8c6096d673e" containerName="kserve-container"
Sep 16 19:25:51.535882 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.535302    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="417f90db-905b-45bd-8cb6-80bb7c7410c8" containerName="kserve-container"
Sep 16 19:25:51.535882 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.535313    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="417f90db-905b-45bd-8cb6-80bb7c7410c8" containerName="kube-rbac-proxy"
Sep 16 19:25:51.535882 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.535323    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1323e73d-f926-45ef-9891-b8c6096d673e" containerName="kserve-container"
Sep 16 19:25:51.535882 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.535334    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1323e73d-f926-45ef-9891-b8c6096d673e" containerName="kube-rbac-proxy"
Sep 16 19:25:51.535882 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.535346    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bf1ae1a-b155-4603-af80-eb136de46752" containerName="kube-rbac-proxy"
Sep 16 19:25:51.535882 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.535356    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bf1ae1a-b155-4603-af80-eb136de46752" containerName="kserve-container"
Sep 16 19:25:51.538455 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.538439    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq"
Sep 16 19:25:51.541201 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.541178    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8gklb\""
Sep 16 19:25:51.541356 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.541280    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:25:51.541356 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.541286    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:25:51.541694 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.541675    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:25:51.541833 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.541711    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:25:51.541897 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.541856    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Sep 16 19:25:51.542027 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.542008    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:25:51.552250 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.552227    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq"]
Sep 16 19:25:51.665326 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.665285    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98p9f\" (UniqueName: \"kubernetes.io/projected/2618a32c-967a-4ec2-bcaf-b31aa4727358-kube-api-access-98p9f\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq\" (UID: \"2618a32c-967a-4ec2-bcaf-b31aa4727358\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq"
Sep 16 19:25:51.665509 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.665343    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2618a32c-967a-4ec2-bcaf-b31aa4727358-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq\" (UID: \"2618a32c-967a-4ec2-bcaf-b31aa4727358\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq"
Sep 16 19:25:51.665509 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.665413    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2618a32c-967a-4ec2-bcaf-b31aa4727358-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq\" (UID: \"2618a32c-967a-4ec2-bcaf-b31aa4727358\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq"
Sep 16 19:25:51.665509 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.665453    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2618a32c-967a-4ec2-bcaf-b31aa4727358-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq\" (UID: \"2618a32c-967a-4ec2-bcaf-b31aa4727358\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq"
Sep 16 19:25:51.665509 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.665483    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2618a32c-967a-4ec2-bcaf-b31aa4727358-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq\" (UID: \"2618a32c-967a-4ec2-bcaf-b31aa4727358\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq"
Sep 16 19:25:51.766681 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.766637    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2618a32c-967a-4ec2-bcaf-b31aa4727358-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq\" (UID: \"2618a32c-967a-4ec2-bcaf-b31aa4727358\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq"
Sep 16 19:25:51.766681 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.766684    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2618a32c-967a-4ec2-bcaf-b31aa4727358-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq\" (UID: \"2618a32c-967a-4ec2-bcaf-b31aa4727358\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq"
Sep 16 19:25:51.766947 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.766762    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2618a32c-967a-4ec2-bcaf-b31aa4727358-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq\" (UID: \"2618a32c-967a-4ec2-bcaf-b31aa4727358\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq"
Sep 16 19:25:51.766947 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.766806    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2618a32c-967a-4ec2-bcaf-b31aa4727358-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq\" (UID: \"2618a32c-967a-4ec2-bcaf-b31aa4727358\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq"
Sep 16 19:25:51.766947 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.766879    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98p9f\" (UniqueName: \"kubernetes.io/projected/2618a32c-967a-4ec2-bcaf-b31aa4727358-kube-api-access-98p9f\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq\" (UID: \"2618a32c-967a-4ec2-bcaf-b31aa4727358\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq"
Sep 16 19:25:51.767324 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.767298    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2618a32c-967a-4ec2-bcaf-b31aa4727358-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq\" (UID: \"2618a32c-967a-4ec2-bcaf-b31aa4727358\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq"
Sep 16 19:25:51.767461 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.767441    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2618a32c-967a-4ec2-bcaf-b31aa4727358-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq\" (UID: \"2618a32c-967a-4ec2-bcaf-b31aa4727358\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq"
Sep 16 19:25:51.767532 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.767488    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2618a32c-967a-4ec2-bcaf-b31aa4727358-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq\" (UID: \"2618a32c-967a-4ec2-bcaf-b31aa4727358\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq"
Sep 16 19:25:51.769436 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.769411    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2618a32c-967a-4ec2-bcaf-b31aa4727358-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq\" (UID: \"2618a32c-967a-4ec2-bcaf-b31aa4727358\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq"
Sep 16 19:25:51.775117 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.775090    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98p9f\" (UniqueName: \"kubernetes.io/projected/2618a32c-967a-4ec2-bcaf-b31aa4727358-kube-api-access-98p9f\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq\" (UID: \"2618a32c-967a-4ec2-bcaf-b31aa4727358\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq"
Sep 16 19:25:51.848780 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:51.848681    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq"
Sep 16 19:25:52.120300 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:52.120274    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq"]
Sep 16 19:25:52.122292 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:25:52.122267    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2618a32c_967a_4ec2_bcaf_b31aa4727358.slice/crio-e6e17f079d6bcdd43f2daf7365e0c71daed3c218225cabef0326401bee3c68e1 WatchSource:0}: Error finding container e6e17f079d6bcdd43f2daf7365e0c71daed3c218225cabef0326401bee3c68e1: Status 404 returned error can't find the container with id e6e17f079d6bcdd43f2daf7365e0c71daed3c218225cabef0326401bee3c68e1
Sep 16 19:25:52.462158 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:52.462068    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq" event={"ID":"2618a32c-967a-4ec2-bcaf-b31aa4727358","Type":"ContainerStarted","Data":"9ccb8fc29ae1b6e45b700e883f6995492a5f58d552ad9a39428342dad67d90f3"}
Sep 16 19:25:52.462158 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:52.462140    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq" event={"ID":"2618a32c-967a-4ec2-bcaf-b31aa4727358","Type":"ContainerStarted","Data":"e6e17f079d6bcdd43f2daf7365e0c71daed3c218225cabef0326401bee3c68e1"}
Sep 16 19:25:56.481472 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:56.481440    2554 generic.go:356] "Generic (PLEG): container finished" podID="2618a32c-967a-4ec2-bcaf-b31aa4727358" containerID="9ccb8fc29ae1b6e45b700e883f6995492a5f58d552ad9a39428342dad67d90f3" exitCode=0
Sep 16 19:25:56.481472 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:56.481475    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq" event={"ID":"2618a32c-967a-4ec2-bcaf-b31aa4727358","Type":"ContainerDied","Data":"9ccb8fc29ae1b6e45b700e883f6995492a5f58d552ad9a39428342dad67d90f3"}
Sep 16 19:25:56.482590 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:25:56.482574    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:26:03.518822 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:03.518785    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq" event={"ID":"2618a32c-967a-4ec2-bcaf-b31aa4727358","Type":"ContainerStarted","Data":"949a23d6dbb7020ff6600a49e58916b7c92d80d803e22ea9a55ae40187faa3d1"}
Sep 16 19:26:03.519283 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:03.518850    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq" event={"ID":"2618a32c-967a-4ec2-bcaf-b31aa4727358","Type":"ContainerStarted","Data":"fc3d8740af2427ddffc968445c2e12d47d38befdeba02de99641a52123cde85a"}
Sep 16 19:26:03.519283 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:03.519062    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq"
Sep 16 19:26:03.545589 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:03.545526    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq" podStartSLOduration=2.602785137 podStartE2EDuration="12.545510732s" totalImagesPullingTime="6.21839288s" totalInitContainerRuntime="3.724332715s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:26:03.543879623 +0000 UTC m=+2473.619649255" watchObservedRunningTime="2026-09-16 19:26:03.545510732 +0000 UTC m=+2473.621280363"
Sep 16 19:26:04.522913 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:04.522881    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq"
Sep 16 19:26:04.524051 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:04.524029    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq" podUID="2618a32c-967a-4ec2-bcaf-b31aa4727358" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:26:05.527587 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:05.527553    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq" podUID="2618a32c-967a-4ec2-bcaf-b31aa4727358" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:26:10.531480 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:10.531452    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq"
Sep 16 19:26:10.531876 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:10.531852    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq" podUID="2618a32c-967a-4ec2-bcaf-b31aa4727358" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:26:20.532318 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:20.532284    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq" podUID="2618a32c-967a-4ec2-bcaf-b31aa4727358" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:26:30.532603 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:30.532564    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq" podUID="2618a32c-967a-4ec2-bcaf-b31aa4727358" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:26:34.977123 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:34.977090    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5"]
Sep 16 19:26:34.983400 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:34.983370    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5"
Sep 16 19:26:34.986063 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:34.986040    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Sep 16 19:26:34.986528 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:34.986509    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:26:34.991828 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:34.991804    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5"]
Sep 16 19:26:35.017044 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:35.017016    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74e70f40-db89-4920-99b0-92a4a77d76a9-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5\" (UID: \"74e70f40-db89-4920-99b0-92a4a77d76a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5"
Sep 16 19:26:35.017148 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:35.017075    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74e70f40-db89-4920-99b0-92a4a77d76a9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5\" (UID: \"74e70f40-db89-4920-99b0-92a4a77d76a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5"
Sep 16 19:26:35.017148 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:35.017105    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kflts\" (UniqueName: \"kubernetes.io/projected/74e70f40-db89-4920-99b0-92a4a77d76a9-kube-api-access-kflts\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5\" (UID: \"74e70f40-db89-4920-99b0-92a4a77d76a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5"
Sep 16 19:26:35.017266 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:35.017154    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74e70f40-db89-4920-99b0-92a4a77d76a9-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5\" (UID: \"74e70f40-db89-4920-99b0-92a4a77d76a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5"
Sep 16 19:26:35.017266 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:35.017180    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74e70f40-db89-4920-99b0-92a4a77d76a9-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5\" (UID: \"74e70f40-db89-4920-99b0-92a4a77d76a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5"
Sep 16 19:26:35.118052 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:35.118029    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74e70f40-db89-4920-99b0-92a4a77d76a9-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5\" (UID: \"74e70f40-db89-4920-99b0-92a4a77d76a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5"
Sep 16 19:26:35.118146 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:35.118076    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74e70f40-db89-4920-99b0-92a4a77d76a9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5\" (UID: \"74e70f40-db89-4920-99b0-92a4a77d76a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5"
Sep 16 19:26:35.118146 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:35.118096    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kflts\" (UniqueName: \"kubernetes.io/projected/74e70f40-db89-4920-99b0-92a4a77d76a9-kube-api-access-kflts\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5\" (UID: \"74e70f40-db89-4920-99b0-92a4a77d76a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5"
Sep 16 19:26:35.118146 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:35.118126    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74e70f40-db89-4920-99b0-92a4a77d76a9-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5\" (UID: \"74e70f40-db89-4920-99b0-92a4a77d76a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5"
Sep 16 19:26:35.118146 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:35.118144    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74e70f40-db89-4920-99b0-92a4a77d76a9-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5\" (UID: \"74e70f40-db89-4920-99b0-92a4a77d76a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5"
Sep 16 19:26:35.118418 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:35.118395    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74e70f40-db89-4920-99b0-92a4a77d76a9-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5\" (UID: \"74e70f40-db89-4920-99b0-92a4a77d76a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5"
Sep 16 19:26:35.118740 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:35.118700    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74e70f40-db89-4920-99b0-92a4a77d76a9-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5\" (UID: \"74e70f40-db89-4920-99b0-92a4a77d76a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5"
Sep 16 19:26:35.118832 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:35.118815    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74e70f40-db89-4920-99b0-92a4a77d76a9-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5\" (UID: \"74e70f40-db89-4920-99b0-92a4a77d76a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5"
Sep 16 19:26:35.120476 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:35.120458    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74e70f40-db89-4920-99b0-92a4a77d76a9-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5\" (UID: \"74e70f40-db89-4920-99b0-92a4a77d76a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5"
Sep 16 19:26:35.126141 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:35.126119    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kflts\" (UniqueName: \"kubernetes.io/projected/74e70f40-db89-4920-99b0-92a4a77d76a9-kube-api-access-kflts\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5\" (UID: \"74e70f40-db89-4920-99b0-92a4a77d76a9\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5"
Sep 16 19:26:35.293915 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:35.293886    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5"
Sep 16 19:26:35.571409 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:35.571387    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5"]
Sep 16 19:26:35.573042 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:26:35.573015    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e70f40_db89_4920_99b0_92a4a77d76a9.slice/crio-42ca60cd5759062fc47f9678756b39e0d40f77b2eb5ad8c43e5786c184c69312 WatchSource:0}: Error finding container 42ca60cd5759062fc47f9678756b39e0d40f77b2eb5ad8c43e5786c184c69312: Status 404 returned error can't find the container with id 42ca60cd5759062fc47f9678756b39e0d40f77b2eb5ad8c43e5786c184c69312
Sep 16 19:26:35.661966 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:35.661937    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5" event={"ID":"74e70f40-db89-4920-99b0-92a4a77d76a9","Type":"ContainerStarted","Data":"c717c0fba289195c2de813726a7e34fa0c50226f0fe2caa23f76fd51c7d03e61"}
Sep 16 19:26:35.662067 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:35.661997    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5" event={"ID":"74e70f40-db89-4920-99b0-92a4a77d76a9","Type":"ContainerStarted","Data":"42ca60cd5759062fc47f9678756b39e0d40f77b2eb5ad8c43e5786c184c69312"}
Sep 16 19:26:39.680787 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:39.680687    2554 generic.go:356] "Generic (PLEG): container finished" podID="74e70f40-db89-4920-99b0-92a4a77d76a9" containerID="c717c0fba289195c2de813726a7e34fa0c50226f0fe2caa23f76fd51c7d03e61" exitCode=0
Sep 16 19:26:39.680787 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:39.680762    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5" event={"ID":"74e70f40-db89-4920-99b0-92a4a77d76a9","Type":"ContainerDied","Data":"c717c0fba289195c2de813726a7e34fa0c50226f0fe2caa23f76fd51c7d03e61"}
Sep 16 19:26:40.532575 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:40.532539    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq" podUID="2618a32c-967a-4ec2-bcaf-b31aa4727358" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:26:40.687127 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:40.687095    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5" event={"ID":"74e70f40-db89-4920-99b0-92a4a77d76a9","Type":"ContainerStarted","Data":"2203d9b8ca8f3b8cf07a9f6945636b066b36c1dd7cee519f89c4331b28720125"}
Sep 16 19:26:40.687497 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:40.687153    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5" event={"ID":"74e70f40-db89-4920-99b0-92a4a77d76a9","Type":"ContainerStarted","Data":"56f2c3d86af218cc32d581086cc0b5252f047942326aa111775df19b28ef4622"}
Sep 16 19:26:40.687497 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:40.687433    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5"
Sep 16 19:26:40.687601 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:40.687504    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5"
Sep 16 19:26:40.705836 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:40.705786    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5" podStartSLOduration=2.9635735050000003 podStartE2EDuration="6.705772747s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.742199242s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:26:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:26:40.705150793 +0000 UTC m=+2510.780920449" watchObservedRunningTime="2026-09-16 19:26:40.705772747 +0000 UTC m=+2510.781542379"
Sep 16 19:26:46.697180 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:46.697110    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5"
Sep 16 19:26:50.532609 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:26:50.532568    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq" podUID="2618a32c-967a-4ec2-bcaf-b31aa4727358" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:27:00.532133 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:00.532092    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq" podUID="2618a32c-967a-4ec2-bcaf-b31aa4727358" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:27:10.532094 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:10.532052    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq" podUID="2618a32c-967a-4ec2-bcaf-b31aa4727358" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Sep 16 19:27:16.751131 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:16.751079    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5" podUID="74e70f40-db89-4920-99b0-92a4a77d76a9" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Sep 16 19:27:20.532883 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:20.532857    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq"
Sep 16 19:27:21.762968 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:21.762934    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq"]
Sep 16 19:27:21.763334 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:21.763273    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq" podUID="2618a32c-967a-4ec2-bcaf-b31aa4727358" containerName="kserve-container" containerID="cri-o://fc3d8740af2427ddffc968445c2e12d47d38befdeba02de99641a52123cde85a" gracePeriod=30
Sep 16 19:27:21.763438 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:21.763369    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq" podUID="2618a32c-967a-4ec2-bcaf-b31aa4727358" containerName="kube-rbac-proxy" containerID="cri-o://949a23d6dbb7020ff6600a49e58916b7c92d80d803e22ea9a55ae40187faa3d1" gracePeriod=30
Sep 16 19:27:22.881816 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:22.881785    2554 generic.go:356] "Generic (PLEG): container finished" podID="2618a32c-967a-4ec2-bcaf-b31aa4727358" containerID="949a23d6dbb7020ff6600a49e58916b7c92d80d803e22ea9a55ae40187faa3d1" exitCode=2
Sep 16 19:27:22.882196 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:22.881833    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq" event={"ID":"2618a32c-967a-4ec2-bcaf-b31aa4727358","Type":"ContainerDied","Data":"949a23d6dbb7020ff6600a49e58916b7c92d80d803e22ea9a55ae40187faa3d1"}
Sep 16 19:27:25.600700 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.600677    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq"
Sep 16 19:27:25.718025 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.717954    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2618a32c-967a-4ec2-bcaf-b31aa4727358-kserve-provision-location\") pod \"2618a32c-967a-4ec2-bcaf-b31aa4727358\" (UID: \"2618a32c-967a-4ec2-bcaf-b31aa4727358\") "
Sep 16 19:27:25.718178 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.718042    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2618a32c-967a-4ec2-bcaf-b31aa4727358-kube-rbac-proxy-sar-config\") pod \"2618a32c-967a-4ec2-bcaf-b31aa4727358\" (UID: \"2618a32c-967a-4ec2-bcaf-b31aa4727358\") "
Sep 16 19:27:25.718178 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.718063    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2618a32c-967a-4ec2-bcaf-b31aa4727358-proxy-tls\") pod \"2618a32c-967a-4ec2-bcaf-b31aa4727358\" (UID: \"2618a32c-967a-4ec2-bcaf-b31aa4727358\") "
Sep 16 19:27:25.718178 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.718078    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2618a32c-967a-4ec2-bcaf-b31aa4727358-cabundle-cert\") pod \"2618a32c-967a-4ec2-bcaf-b31aa4727358\" (UID: \"2618a32c-967a-4ec2-bcaf-b31aa4727358\") "
Sep 16 19:27:25.718178 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.718103    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98p9f\" (UniqueName: \"kubernetes.io/projected/2618a32c-967a-4ec2-bcaf-b31aa4727358-kube-api-access-98p9f\") pod \"2618a32c-967a-4ec2-bcaf-b31aa4727358\" (UID: \"2618a32c-967a-4ec2-bcaf-b31aa4727358\") "
Sep 16 19:27:25.718403 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.718297    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2618a32c-967a-4ec2-bcaf-b31aa4727358-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2618a32c-967a-4ec2-bcaf-b31aa4727358" (UID: "2618a32c-967a-4ec2-bcaf-b31aa4727358"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:25.718460 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.718405    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2618a32c-967a-4ec2-bcaf-b31aa4727358-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2618a32c-967a-4ec2-bcaf-b31aa4727358" (UID: "2618a32c-967a-4ec2-bcaf-b31aa4727358"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:27:25.718460 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.718439    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2618a32c-967a-4ec2-bcaf-b31aa4727358-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2618a32c-967a-4ec2-bcaf-b31aa4727358" (UID: "2618a32c-967a-4ec2-bcaf-b31aa4727358"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:27:25.720200 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.720180    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2618a32c-967a-4ec2-bcaf-b31aa4727358-kube-api-access-98p9f" (OuterVolumeSpecName: "kube-api-access-98p9f") pod "2618a32c-967a-4ec2-bcaf-b31aa4727358" (UID: "2618a32c-967a-4ec2-bcaf-b31aa4727358"). InnerVolumeSpecName "kube-api-access-98p9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:27:25.720280 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.720267    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2618a32c-967a-4ec2-bcaf-b31aa4727358-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2618a32c-967a-4ec2-bcaf-b31aa4727358" (UID: "2618a32c-967a-4ec2-bcaf-b31aa4727358"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:27:25.819188 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.819166    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2618a32c-967a-4ec2-bcaf-b31aa4727358-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:27:25.819188 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.819186    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2618a32c-967a-4ec2-bcaf-b31aa4727358-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:27:25.819315 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.819197    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2618a32c-967a-4ec2-bcaf-b31aa4727358-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:27:25.819315 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.819205    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98p9f\" (UniqueName: \"kubernetes.io/projected/2618a32c-967a-4ec2-bcaf-b31aa4727358-kube-api-access-98p9f\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:27:25.819315 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.819214    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2618a32c-967a-4ec2-bcaf-b31aa4727358-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:27:25.898048 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.898014    2554 generic.go:356] "Generic (PLEG): container finished" podID="2618a32c-967a-4ec2-bcaf-b31aa4727358" containerID="fc3d8740af2427ddffc968445c2e12d47d38befdeba02de99641a52123cde85a" exitCode=0
Sep 16 19:27:25.898156 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.898063    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq" event={"ID":"2618a32c-967a-4ec2-bcaf-b31aa4727358","Type":"ContainerDied","Data":"fc3d8740af2427ddffc968445c2e12d47d38befdeba02de99641a52123cde85a"}
Sep 16 19:27:25.898156 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.898078    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq"
Sep 16 19:27:25.898156 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.898108    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq" event={"ID":"2618a32c-967a-4ec2-bcaf-b31aa4727358","Type":"ContainerDied","Data":"e6e17f079d6bcdd43f2daf7365e0c71daed3c218225cabef0326401bee3c68e1"}
Sep 16 19:27:25.898156 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.898134    2554 scope.go:117] "RemoveContainer" containerID="949a23d6dbb7020ff6600a49e58916b7c92d80d803e22ea9a55ae40187faa3d1"
Sep 16 19:27:25.907435 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.907420    2554 scope.go:117] "RemoveContainer" containerID="fc3d8740af2427ddffc968445c2e12d47d38befdeba02de99641a52123cde85a"
Sep 16 19:27:25.914745 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.914730    2554 scope.go:117] "RemoveContainer" containerID="9ccb8fc29ae1b6e45b700e883f6995492a5f58d552ad9a39428342dad67d90f3"
Sep 16 19:27:25.921045 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.921021    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq"]
Sep 16 19:27:25.923280 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.923267    2554 scope.go:117] "RemoveContainer" containerID="949a23d6dbb7020ff6600a49e58916b7c92d80d803e22ea9a55ae40187faa3d1"
Sep 16 19:27:25.923544 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:27:25.923526    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"949a23d6dbb7020ff6600a49e58916b7c92d80d803e22ea9a55ae40187faa3d1\": container with ID starting with 949a23d6dbb7020ff6600a49e58916b7c92d80d803e22ea9a55ae40187faa3d1 not found: ID does not exist" containerID="949a23d6dbb7020ff6600a49e58916b7c92d80d803e22ea9a55ae40187faa3d1"
Sep 16 19:27:25.923604 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.923550    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"949a23d6dbb7020ff6600a49e58916b7c92d80d803e22ea9a55ae40187faa3d1"} err="failed to get container status \"949a23d6dbb7020ff6600a49e58916b7c92d80d803e22ea9a55ae40187faa3d1\": rpc error: code = NotFound desc = could not find container \"949a23d6dbb7020ff6600a49e58916b7c92d80d803e22ea9a55ae40187faa3d1\": container with ID starting with 949a23d6dbb7020ff6600a49e58916b7c92d80d803e22ea9a55ae40187faa3d1 not found: ID does not exist"
Sep 16 19:27:25.923604 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.923590    2554 scope.go:117] "RemoveContainer" containerID="fc3d8740af2427ddffc968445c2e12d47d38befdeba02de99641a52123cde85a"
Sep 16 19:27:25.923886 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:27:25.923862    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc3d8740af2427ddffc968445c2e12d47d38befdeba02de99641a52123cde85a\": container with ID starting with fc3d8740af2427ddffc968445c2e12d47d38befdeba02de99641a52123cde85a not found: ID does not exist" containerID="fc3d8740af2427ddffc968445c2e12d47d38befdeba02de99641a52123cde85a"
Sep 16 19:27:25.923978 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.923894    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc3d8740af2427ddffc968445c2e12d47d38befdeba02de99641a52123cde85a"} err="failed to get container status \"fc3d8740af2427ddffc968445c2e12d47d38befdeba02de99641a52123cde85a\": rpc error: code = NotFound desc = could not find container \"fc3d8740af2427ddffc968445c2e12d47d38befdeba02de99641a52123cde85a\": container with ID starting with fc3d8740af2427ddffc968445c2e12d47d38befdeba02de99641a52123cde85a not found: ID does not exist"
Sep 16 19:27:25.923978 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.923914    2554 scope.go:117] "RemoveContainer" containerID="9ccb8fc29ae1b6e45b700e883f6995492a5f58d552ad9a39428342dad67d90f3"
Sep 16 19:27:25.924182 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:27:25.924164    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ccb8fc29ae1b6e45b700e883f6995492a5f58d552ad9a39428342dad67d90f3\": container with ID starting with 9ccb8fc29ae1b6e45b700e883f6995492a5f58d552ad9a39428342dad67d90f3 not found: ID does not exist" containerID="9ccb8fc29ae1b6e45b700e883f6995492a5f58d552ad9a39428342dad67d90f3"
Sep 16 19:27:25.924245 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.924199    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ccb8fc29ae1b6e45b700e883f6995492a5f58d552ad9a39428342dad67d90f3"} err="failed to get container status \"9ccb8fc29ae1b6e45b700e883f6995492a5f58d552ad9a39428342dad67d90f3\": rpc error: code = NotFound desc = could not find container \"9ccb8fc29ae1b6e45b700e883f6995492a5f58d552ad9a39428342dad67d90f3\": container with ID starting with 9ccb8fc29ae1b6e45b700e883f6995492a5f58d552ad9a39428342dad67d90f3 not found: ID does not exist"
Sep 16 19:27:25.926208 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:25.926190    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq"]
Sep 16 19:27:26.443415 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:26.443386    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2618a32c-967a-4ec2-bcaf-b31aa4727358" path="/var/lib/kubelet/pods/2618a32c-967a-4ec2-bcaf-b31aa4727358/volumes"
Sep 16 19:27:26.528280 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:26.528250    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-dd2sq" podUID="2618a32c-967a-4ec2-bcaf-b31aa4727358" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:27:26.699649 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:26.699584    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5"
Sep 16 19:27:26.977057 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:26.976974    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"]
Sep 16 19:27:26.977409 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:26.977390    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2618a32c-967a-4ec2-bcaf-b31aa4727358" containerName="kserve-container"
Sep 16 19:27:26.977501 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:26.977413    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2618a32c-967a-4ec2-bcaf-b31aa4727358" containerName="kserve-container"
Sep 16 19:27:26.977501 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:26.977427    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2618a32c-967a-4ec2-bcaf-b31aa4727358" containerName="kube-rbac-proxy"
Sep 16 19:27:26.977501 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:26.977435    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2618a32c-967a-4ec2-bcaf-b31aa4727358" containerName="kube-rbac-proxy"
Sep 16 19:27:26.977501 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:26.977451    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2618a32c-967a-4ec2-bcaf-b31aa4727358" containerName="storage-initializer"
Sep 16 19:27:26.977501 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:26.977460    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2618a32c-967a-4ec2-bcaf-b31aa4727358" containerName="storage-initializer"
Sep 16 19:27:26.977792 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:26.977542    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2618a32c-967a-4ec2-bcaf-b31aa4727358" containerName="kserve-container"
Sep 16 19:27:26.977792 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:26.977556    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2618a32c-967a-4ec2-bcaf-b31aa4727358" containerName="kube-rbac-proxy"
Sep 16 19:27:26.984095 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:26.984066    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"
Sep 16 19:27:26.992008 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:26.991991    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Sep 16 19:27:26.992525 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:26.992513    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:27:27.013441 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:27.013417    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"]
Sep 16 19:27:27.030107 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:27.030083    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a8a04b7-1602-4f52-99c7-376443de5301-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs\" (UID: \"7a8a04b7-1602-4f52-99c7-376443de5301\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"
Sep 16 19:27:27.030107 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:27.030108    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a8a04b7-1602-4f52-99c7-376443de5301-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs\" (UID: \"7a8a04b7-1602-4f52-99c7-376443de5301\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"
Sep 16 19:27:27.030246 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:27.030154    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j69pd\" (UniqueName: \"kubernetes.io/projected/7a8a04b7-1602-4f52-99c7-376443de5301-kube-api-access-j69pd\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs\" (UID: \"7a8a04b7-1602-4f52-99c7-376443de5301\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"
Sep 16 19:27:27.030246 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:27.030194    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a8a04b7-1602-4f52-99c7-376443de5301-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs\" (UID: \"7a8a04b7-1602-4f52-99c7-376443de5301\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"
Sep 16 19:27:27.030246 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:27.030217    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a8a04b7-1602-4f52-99c7-376443de5301-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs\" (UID: \"7a8a04b7-1602-4f52-99c7-376443de5301\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"
Sep 16 19:27:27.130656 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:27.130633    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a8a04b7-1602-4f52-99c7-376443de5301-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs\" (UID: \"7a8a04b7-1602-4f52-99c7-376443de5301\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"
Sep 16 19:27:27.130771 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:27.130659    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a8a04b7-1602-4f52-99c7-376443de5301-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs\" (UID: \"7a8a04b7-1602-4f52-99c7-376443de5301\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"
Sep 16 19:27:27.130771 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:27.130710    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j69pd\" (UniqueName: \"kubernetes.io/projected/7a8a04b7-1602-4f52-99c7-376443de5301-kube-api-access-j69pd\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs\" (UID: \"7a8a04b7-1602-4f52-99c7-376443de5301\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"
Sep 16 19:27:27.130880 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:27.130770    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a8a04b7-1602-4f52-99c7-376443de5301-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs\" (UID: \"7a8a04b7-1602-4f52-99c7-376443de5301\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"
Sep 16 19:27:27.130880 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:27:27.130798    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-serving-cert: secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Sep 16 19:27:27.130880 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:27.130805    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a8a04b7-1602-4f52-99c7-376443de5301-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs\" (UID: \"7a8a04b7-1602-4f52-99c7-376443de5301\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"
Sep 16 19:27:27.130880 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:27:27.130867    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a8a04b7-1602-4f52-99c7-376443de5301-proxy-tls podName:7a8a04b7-1602-4f52-99c7-376443de5301 nodeName:}" failed. No retries permitted until 2026-09-16 19:27:27.63084826 +0000 UTC m=+2557.706617884 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7a8a04b7-1602-4f52-99c7-376443de5301-proxy-tls") pod "isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs" (UID: "7a8a04b7-1602-4f52-99c7-376443de5301") : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Sep 16 19:27:27.131220 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:27.131201    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a8a04b7-1602-4f52-99c7-376443de5301-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs\" (UID: \"7a8a04b7-1602-4f52-99c7-376443de5301\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"
Sep 16 19:27:27.131383 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:27.131366    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a8a04b7-1602-4f52-99c7-376443de5301-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs\" (UID: \"7a8a04b7-1602-4f52-99c7-376443de5301\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"
Sep 16 19:27:27.131422 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:27.131390    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a8a04b7-1602-4f52-99c7-376443de5301-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs\" (UID: \"7a8a04b7-1602-4f52-99c7-376443de5301\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"
Sep 16 19:27:27.141523 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:27.141502    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j69pd\" (UniqueName: \"kubernetes.io/projected/7a8a04b7-1602-4f52-99c7-376443de5301-kube-api-access-j69pd\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs\" (UID: \"7a8a04b7-1602-4f52-99c7-376443de5301\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"
Sep 16 19:27:27.634048 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:27.634018    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a8a04b7-1602-4f52-99c7-376443de5301-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs\" (UID: \"7a8a04b7-1602-4f52-99c7-376443de5301\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"
Sep 16 19:27:27.636468 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:27.636444    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a8a04b7-1602-4f52-99c7-376443de5301-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs\" (UID: \"7a8a04b7-1602-4f52-99c7-376443de5301\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"
Sep 16 19:27:27.895066 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:27.894996    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"
Sep 16 19:27:28.173914 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:28.173887    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"]
Sep 16 19:27:28.175536 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:27:28.175511    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a8a04b7_1602_4f52_99c7_376443de5301.slice/crio-4334e5903771f8a6e7812ad97b74cc5d88901f48d81b10ba9233c4c58769232d WatchSource:0}: Error finding container 4334e5903771f8a6e7812ad97b74cc5d88901f48d81b10ba9233c4c58769232d: Status 404 returned error can't find the container with id 4334e5903771f8a6e7812ad97b74cc5d88901f48d81b10ba9233c4c58769232d
Sep 16 19:27:28.916700 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:28.916660    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs" event={"ID":"7a8a04b7-1602-4f52-99c7-376443de5301","Type":"ContainerStarted","Data":"ffa63de3cfdb0fc1272350226d3bbe4a21b249241023df481689767c2695953e"}
Sep 16 19:27:28.917082 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:28.916758    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs" event={"ID":"7a8a04b7-1602-4f52-99c7-376443de5301","Type":"ContainerStarted","Data":"4334e5903771f8a6e7812ad97b74cc5d88901f48d81b10ba9233c4c58769232d"}
Sep 16 19:27:32.938045 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:32.938012    2554 generic.go:356] "Generic (PLEG): container finished" podID="7a8a04b7-1602-4f52-99c7-376443de5301" containerID="ffa63de3cfdb0fc1272350226d3bbe4a21b249241023df481689767c2695953e" exitCode=0
Sep 16 19:27:32.938398 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:32.938067    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs" event={"ID":"7a8a04b7-1602-4f52-99c7-376443de5301","Type":"ContainerDied","Data":"ffa63de3cfdb0fc1272350226d3bbe4a21b249241023df481689767c2695953e"}
Sep 16 19:27:33.945487 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:33.945451    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs" event={"ID":"7a8a04b7-1602-4f52-99c7-376443de5301","Type":"ContainerStarted","Data":"bb7ecb377045ea5136b5bbe5bcf58984090d548b7f036d4ad2d4802094124f21"}
Sep 16 19:27:33.945865 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:33.945517    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs" event={"ID":"7a8a04b7-1602-4f52-99c7-376443de5301","Type":"ContainerStarted","Data":"fca8126d35a89d5fd5b76ffde016e7b27d1dfee8f86a34dad18c02b94ffa20cb"}
Sep 16 19:27:33.945865 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:33.945826    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"
Sep 16 19:27:33.945944 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:33.945884    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"
Sep 16 19:27:33.969589 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:33.969543    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs" podStartSLOduration=4.116640572 podStartE2EDuration="7.969529365s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.852888793s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:27:33.967364212 +0000 UTC m=+2564.043133865" watchObservedRunningTime="2026-09-16 19:27:33.969529365 +0000 UTC m=+2564.045298997"
Sep 16 19:27:35.043536 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:35.043505    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5"]
Sep 16 19:27:35.043962 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:35.043737    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5" podUID="74e70f40-db89-4920-99b0-92a4a77d76a9" containerName="kserve-container" containerID="cri-o://56f2c3d86af218cc32d581086cc0b5252f047942326aa111775df19b28ef4622" gracePeriod=30
Sep 16 19:27:35.043962 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:35.043769    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5" podUID="74e70f40-db89-4920-99b0-92a4a77d76a9" containerName="kube-rbac-proxy" containerID="cri-o://2203d9b8ca8f3b8cf07a9f6945636b066b36c1dd7cee519f89c4331b28720125" gracePeriod=30
Sep 16 19:27:35.960541 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:35.960502    2554 generic.go:356] "Generic (PLEG): container finished" podID="74e70f40-db89-4920-99b0-92a4a77d76a9" containerID="2203d9b8ca8f3b8cf07a9f6945636b066b36c1dd7cee519f89c4331b28720125" exitCode=2
Sep 16 19:27:35.960541 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:35.960527    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5" event={"ID":"74e70f40-db89-4920-99b0-92a4a77d76a9","Type":"ContainerDied","Data":"2203d9b8ca8f3b8cf07a9f6945636b066b36c1dd7cee519f89c4331b28720125"}
Sep 16 19:27:36.692365 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:36.692328    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5" podUID="74e70f40-db89-4920-99b0-92a4a77d76a9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.58:8643/healthz\": dial tcp 10.134.0.58:8643: connect: connection refused"
Sep 16 19:27:37.738875 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:37.738832    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5" podUID="74e70f40-db89-4920-99b0-92a4a77d76a9" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.58:8080/v2/models/isvc-xgboost-v2-runtime/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 16 19:27:39.957442 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:39.957409    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"
Sep 16 19:27:41.281570 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:41.281547    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5"
Sep 16 19:27:41.341770 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:41.341680    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74e70f40-db89-4920-99b0-92a4a77d76a9-kserve-provision-location\") pod \"74e70f40-db89-4920-99b0-92a4a77d76a9\" (UID: \"74e70f40-db89-4920-99b0-92a4a77d76a9\") "
Sep 16 19:27:41.341770 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:41.341735    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74e70f40-db89-4920-99b0-92a4a77d76a9-proxy-tls\") pod \"74e70f40-db89-4920-99b0-92a4a77d76a9\" (UID: \"74e70f40-db89-4920-99b0-92a4a77d76a9\") "
Sep 16 19:27:41.341770 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:41.341761    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74e70f40-db89-4920-99b0-92a4a77d76a9-cabundle-cert\") pod \"74e70f40-db89-4920-99b0-92a4a77d76a9\" (UID: \"74e70f40-db89-4920-99b0-92a4a77d76a9\") "
Sep 16 19:27:41.342025 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:41.341814    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74e70f40-db89-4920-99b0-92a4a77d76a9-kube-rbac-proxy-sar-config\") pod \"74e70f40-db89-4920-99b0-92a4a77d76a9\" (UID: \"74e70f40-db89-4920-99b0-92a4a77d76a9\") "
Sep 16 19:27:41.342025 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:41.341873    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kflts\" (UniqueName: \"kubernetes.io/projected/74e70f40-db89-4920-99b0-92a4a77d76a9-kube-api-access-kflts\") pod \"74e70f40-db89-4920-99b0-92a4a77d76a9\" (UID: \"74e70f40-db89-4920-99b0-92a4a77d76a9\") "
Sep 16 19:27:41.342127 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:41.342035    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74e70f40-db89-4920-99b0-92a4a77d76a9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "74e70f40-db89-4920-99b0-92a4a77d76a9" (UID: "74e70f40-db89-4920-99b0-92a4a77d76a9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:27:41.342175 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:41.342137    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74e70f40-db89-4920-99b0-92a4a77d76a9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "74e70f40-db89-4920-99b0-92a4a77d76a9" (UID: "74e70f40-db89-4920-99b0-92a4a77d76a9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:27:41.342266 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:41.342243    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74e70f40-db89-4920-99b0-92a4a77d76a9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "74e70f40-db89-4920-99b0-92a4a77d76a9" (UID: "74e70f40-db89-4920-99b0-92a4a77d76a9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:27:41.343759 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:41.343734    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74e70f40-db89-4920-99b0-92a4a77d76a9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "74e70f40-db89-4920-99b0-92a4a77d76a9" (UID: "74e70f40-db89-4920-99b0-92a4a77d76a9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:27:41.343876 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:41.343848    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74e70f40-db89-4920-99b0-92a4a77d76a9-kube-api-access-kflts" (OuterVolumeSpecName: "kube-api-access-kflts") pod "74e70f40-db89-4920-99b0-92a4a77d76a9" (UID: "74e70f40-db89-4920-99b0-92a4a77d76a9"). InnerVolumeSpecName "kube-api-access-kflts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:27:41.442981 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:41.442959    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74e70f40-db89-4920-99b0-92a4a77d76a9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:27:41.442981 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:41.442980    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kflts\" (UniqueName: \"kubernetes.io/projected/74e70f40-db89-4920-99b0-92a4a77d76a9-kube-api-access-kflts\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:27:41.443124 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:41.442989    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74e70f40-db89-4920-99b0-92a4a77d76a9-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:27:41.443124 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:41.443000    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74e70f40-db89-4920-99b0-92a4a77d76a9-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:27:41.443124 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:41.443009    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74e70f40-db89-4920-99b0-92a4a77d76a9-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:27:41.990653 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:41.990619    2554 generic.go:356] "Generic (PLEG): container finished" podID="74e70f40-db89-4920-99b0-92a4a77d76a9" containerID="56f2c3d86af218cc32d581086cc0b5252f047942326aa111775df19b28ef4622" exitCode=0
Sep 16 19:27:41.990846 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:41.990661    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5" event={"ID":"74e70f40-db89-4920-99b0-92a4a77d76a9","Type":"ContainerDied","Data":"56f2c3d86af218cc32d581086cc0b5252f047942326aa111775df19b28ef4622"}
Sep 16 19:27:41.990846 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:41.990680    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5"
Sep 16 19:27:41.990846 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:41.990707    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5" event={"ID":"74e70f40-db89-4920-99b0-92a4a77d76a9","Type":"ContainerDied","Data":"42ca60cd5759062fc47f9678756b39e0d40f77b2eb5ad8c43e5786c184c69312"}
Sep 16 19:27:41.990846 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:41.990753    2554 scope.go:117] "RemoveContainer" containerID="2203d9b8ca8f3b8cf07a9f6945636b066b36c1dd7cee519f89c4331b28720125"
Sep 16 19:27:41.999532 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:41.999517    2554 scope.go:117] "RemoveContainer" containerID="56f2c3d86af218cc32d581086cc0b5252f047942326aa111775df19b28ef4622"
Sep 16 19:27:42.007004 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:42.006986    2554 scope.go:117] "RemoveContainer" containerID="c717c0fba289195c2de813726a7e34fa0c50226f0fe2caa23f76fd51c7d03e61"
Sep 16 19:27:42.014076 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:42.014057    2554 scope.go:117] "RemoveContainer" containerID="2203d9b8ca8f3b8cf07a9f6945636b066b36c1dd7cee519f89c4331b28720125"
Sep 16 19:27:42.014331 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:27:42.014312    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2203d9b8ca8f3b8cf07a9f6945636b066b36c1dd7cee519f89c4331b28720125\": container with ID starting with 2203d9b8ca8f3b8cf07a9f6945636b066b36c1dd7cee519f89c4331b28720125 not found: ID does not exist" containerID="2203d9b8ca8f3b8cf07a9f6945636b066b36c1dd7cee519f89c4331b28720125"
Sep 16 19:27:42.014389 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:42.014336    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2203d9b8ca8f3b8cf07a9f6945636b066b36c1dd7cee519f89c4331b28720125"} err="failed to get container status \"2203d9b8ca8f3b8cf07a9f6945636b066b36c1dd7cee519f89c4331b28720125\": rpc error: code = NotFound desc = could not find container \"2203d9b8ca8f3b8cf07a9f6945636b066b36c1dd7cee519f89c4331b28720125\": container with ID starting with 2203d9b8ca8f3b8cf07a9f6945636b066b36c1dd7cee519f89c4331b28720125 not found: ID does not exist"
Sep 16 19:27:42.014389 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:42.014352    2554 scope.go:117] "RemoveContainer" containerID="56f2c3d86af218cc32d581086cc0b5252f047942326aa111775df19b28ef4622"
Sep 16 19:27:42.014578 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:27:42.014558    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56f2c3d86af218cc32d581086cc0b5252f047942326aa111775df19b28ef4622\": container with ID starting with 56f2c3d86af218cc32d581086cc0b5252f047942326aa111775df19b28ef4622 not found: ID does not exist" containerID="56f2c3d86af218cc32d581086cc0b5252f047942326aa111775df19b28ef4622"
Sep 16 19:27:42.014620 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:42.014585    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56f2c3d86af218cc32d581086cc0b5252f047942326aa111775df19b28ef4622"} err="failed to get container status \"56f2c3d86af218cc32d581086cc0b5252f047942326aa111775df19b28ef4622\": rpc error: code = NotFound desc = could not find container \"56f2c3d86af218cc32d581086cc0b5252f047942326aa111775df19b28ef4622\": container with ID starting with 56f2c3d86af218cc32d581086cc0b5252f047942326aa111775df19b28ef4622 not found: ID does not exist"
Sep 16 19:27:42.014620 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:42.014602    2554 scope.go:117] "RemoveContainer" containerID="c717c0fba289195c2de813726a7e34fa0c50226f0fe2caa23f76fd51c7d03e61"
Sep 16 19:27:42.014862 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:27:42.014847    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c717c0fba289195c2de813726a7e34fa0c50226f0fe2caa23f76fd51c7d03e61\": container with ID starting with c717c0fba289195c2de813726a7e34fa0c50226f0fe2caa23f76fd51c7d03e61 not found: ID does not exist" containerID="c717c0fba289195c2de813726a7e34fa0c50226f0fe2caa23f76fd51c7d03e61"
Sep 16 19:27:42.014928 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:42.014863    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c717c0fba289195c2de813726a7e34fa0c50226f0fe2caa23f76fd51c7d03e61"} err="failed to get container status \"c717c0fba289195c2de813726a7e34fa0c50226f0fe2caa23f76fd51c7d03e61\": rpc error: code = NotFound desc = could not find container \"c717c0fba289195c2de813726a7e34fa0c50226f0fe2caa23f76fd51c7d03e61\": container with ID starting with c717c0fba289195c2de813726a7e34fa0c50226f0fe2caa23f76fd51c7d03e61 not found: ID does not exist"
Sep 16 19:27:42.030011 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:42.029991    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5"]
Sep 16 19:27:42.033399 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:42.033378    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-tzxb5"]
Sep 16 19:27:42.443572 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:27:42.443547    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74e70f40-db89-4920-99b0-92a4a77d76a9" path="/var/lib/kubelet/pods/74e70f40-db89-4920-99b0-92a4a77d76a9/volumes"
Sep 16 19:28:09.961437 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:09.961408    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"
Sep 16 19:28:17.057230 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:17.057155    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"]
Sep 16 19:28:17.057615 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:17.057374    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs" podUID="7a8a04b7-1602-4f52-99c7-376443de5301" containerName="kserve-container" containerID="cri-o://fca8126d35a89d5fd5b76ffde016e7b27d1dfee8f86a34dad18c02b94ffa20cb" gracePeriod=30
Sep 16 19:28:17.057615 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:17.057412    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs" podUID="7a8a04b7-1602-4f52-99c7-376443de5301" containerName="kube-rbac-proxy" containerID="cri-o://bb7ecb377045ea5136b5bbe5bcf58984090d548b7f036d4ad2d4802094124f21" gracePeriod=30
Sep 16 19:28:18.005146 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.005123    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"
Sep 16 19:28:18.124938 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.124885    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a8a04b7-1602-4f52-99c7-376443de5301-cabundle-cert\") pod \"7a8a04b7-1602-4f52-99c7-376443de5301\" (UID: \"7a8a04b7-1602-4f52-99c7-376443de5301\") "
Sep 16 19:28:18.124938 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.124916    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a8a04b7-1602-4f52-99c7-376443de5301-kube-rbac-proxy-sar-config\") pod \"7a8a04b7-1602-4f52-99c7-376443de5301\" (UID: \"7a8a04b7-1602-4f52-99c7-376443de5301\") "
Sep 16 19:28:18.124938 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.124935    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a8a04b7-1602-4f52-99c7-376443de5301-kserve-provision-location\") pod \"7a8a04b7-1602-4f52-99c7-376443de5301\" (UID: \"7a8a04b7-1602-4f52-99c7-376443de5301\") "
Sep 16 19:28:18.125394 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.124970    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a8a04b7-1602-4f52-99c7-376443de5301-proxy-tls\") pod \"7a8a04b7-1602-4f52-99c7-376443de5301\" (UID: \"7a8a04b7-1602-4f52-99c7-376443de5301\") "
Sep 16 19:28:18.125394 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.124998    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j69pd\" (UniqueName: \"kubernetes.io/projected/7a8a04b7-1602-4f52-99c7-376443de5301-kube-api-access-j69pd\") pod \"7a8a04b7-1602-4f52-99c7-376443de5301\" (UID: \"7a8a04b7-1602-4f52-99c7-376443de5301\") "
Sep 16 19:28:18.125394 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.125258    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a8a04b7-1602-4f52-99c7-376443de5301-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7a8a04b7-1602-4f52-99c7-376443de5301" (UID: "7a8a04b7-1602-4f52-99c7-376443de5301"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:28:18.125394 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.125281    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a8a04b7-1602-4f52-99c7-376443de5301-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7a8a04b7-1602-4f52-99c7-376443de5301" (UID: "7a8a04b7-1602-4f52-99c7-376443de5301"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:28:18.125394 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.125294    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a8a04b7-1602-4f52-99c7-376443de5301-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7a8a04b7-1602-4f52-99c7-376443de5301" (UID: "7a8a04b7-1602-4f52-99c7-376443de5301"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:28:18.126916 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.126896    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a8a04b7-1602-4f52-99c7-376443de5301-kube-api-access-j69pd" (OuterVolumeSpecName: "kube-api-access-j69pd") pod "7a8a04b7-1602-4f52-99c7-376443de5301" (UID: "7a8a04b7-1602-4f52-99c7-376443de5301"). InnerVolumeSpecName "kube-api-access-j69pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:28:18.127061 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.127040    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a8a04b7-1602-4f52-99c7-376443de5301-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7a8a04b7-1602-4f52-99c7-376443de5301" (UID: "7a8a04b7-1602-4f52-99c7-376443de5301"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:28:18.155004 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.154971    2554 generic.go:356] "Generic (PLEG): container finished" podID="7a8a04b7-1602-4f52-99c7-376443de5301" containerID="bb7ecb377045ea5136b5bbe5bcf58984090d548b7f036d4ad2d4802094124f21" exitCode=2
Sep 16 19:28:18.155004 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.154991    2554 generic.go:356] "Generic (PLEG): container finished" podID="7a8a04b7-1602-4f52-99c7-376443de5301" containerID="fca8126d35a89d5fd5b76ffde016e7b27d1dfee8f86a34dad18c02b94ffa20cb" exitCode=0
Sep 16 19:28:18.155182 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.155036    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs" event={"ID":"7a8a04b7-1602-4f52-99c7-376443de5301","Type":"ContainerDied","Data":"bb7ecb377045ea5136b5bbe5bcf58984090d548b7f036d4ad2d4802094124f21"}
Sep 16 19:28:18.155182 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.155081    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs" event={"ID":"7a8a04b7-1602-4f52-99c7-376443de5301","Type":"ContainerDied","Data":"fca8126d35a89d5fd5b76ffde016e7b27d1dfee8f86a34dad18c02b94ffa20cb"}
Sep 16 19:28:18.155182 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.155080    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"
Sep 16 19:28:18.155182 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.155110    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs" event={"ID":"7a8a04b7-1602-4f52-99c7-376443de5301","Type":"ContainerDied","Data":"4334e5903771f8a6e7812ad97b74cc5d88901f48d81b10ba9233c4c58769232d"}
Sep 16 19:28:18.155182 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.155137    2554 scope.go:117] "RemoveContainer" containerID="bb7ecb377045ea5136b5bbe5bcf58984090d548b7f036d4ad2d4802094124f21"
Sep 16 19:28:18.166022 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.165870    2554 scope.go:117] "RemoveContainer" containerID="fca8126d35a89d5fd5b76ffde016e7b27d1dfee8f86a34dad18c02b94ffa20cb"
Sep 16 19:28:18.173912 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.173897    2554 scope.go:117] "RemoveContainer" containerID="ffa63de3cfdb0fc1272350226d3bbe4a21b249241023df481689767c2695953e"
Sep 16 19:28:18.181517 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.181501    2554 scope.go:117] "RemoveContainer" containerID="bb7ecb377045ea5136b5bbe5bcf58984090d548b7f036d4ad2d4802094124f21"
Sep 16 19:28:18.181833 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:28:18.181812    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb7ecb377045ea5136b5bbe5bcf58984090d548b7f036d4ad2d4802094124f21\": container with ID starting with bb7ecb377045ea5136b5bbe5bcf58984090d548b7f036d4ad2d4802094124f21 not found: ID does not exist" containerID="bb7ecb377045ea5136b5bbe5bcf58984090d548b7f036d4ad2d4802094124f21"
Sep 16 19:28:18.181900 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.181841    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb7ecb377045ea5136b5bbe5bcf58984090d548b7f036d4ad2d4802094124f21"} err="failed to get container status \"bb7ecb377045ea5136b5bbe5bcf58984090d548b7f036d4ad2d4802094124f21\": rpc error: code = NotFound desc = could not find container \"bb7ecb377045ea5136b5bbe5bcf58984090d548b7f036d4ad2d4802094124f21\": container with ID starting with bb7ecb377045ea5136b5bbe5bcf58984090d548b7f036d4ad2d4802094124f21 not found: ID does not exist"
Sep 16 19:28:18.181900 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.181858    2554 scope.go:117] "RemoveContainer" containerID="fca8126d35a89d5fd5b76ffde016e7b27d1dfee8f86a34dad18c02b94ffa20cb"
Sep 16 19:28:18.182145 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:28:18.182127    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fca8126d35a89d5fd5b76ffde016e7b27d1dfee8f86a34dad18c02b94ffa20cb\": container with ID starting with fca8126d35a89d5fd5b76ffde016e7b27d1dfee8f86a34dad18c02b94ffa20cb not found: ID does not exist" containerID="fca8126d35a89d5fd5b76ffde016e7b27d1dfee8f86a34dad18c02b94ffa20cb"
Sep 16 19:28:18.182197 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.182150    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca8126d35a89d5fd5b76ffde016e7b27d1dfee8f86a34dad18c02b94ffa20cb"} err="failed to get container status \"fca8126d35a89d5fd5b76ffde016e7b27d1dfee8f86a34dad18c02b94ffa20cb\": rpc error: code = NotFound desc = could not find container \"fca8126d35a89d5fd5b76ffde016e7b27d1dfee8f86a34dad18c02b94ffa20cb\": container with ID starting with fca8126d35a89d5fd5b76ffde016e7b27d1dfee8f86a34dad18c02b94ffa20cb not found: ID does not exist"
Sep 16 19:28:18.182197 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.182164    2554 scope.go:117] "RemoveContainer" containerID="ffa63de3cfdb0fc1272350226d3bbe4a21b249241023df481689767c2695953e"
Sep 16 19:28:18.182391 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:28:18.182375    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffa63de3cfdb0fc1272350226d3bbe4a21b249241023df481689767c2695953e\": container with ID starting with ffa63de3cfdb0fc1272350226d3bbe4a21b249241023df481689767c2695953e not found: ID does not exist" containerID="ffa63de3cfdb0fc1272350226d3bbe4a21b249241023df481689767c2695953e"
Sep 16 19:28:18.182449 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.182394    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffa63de3cfdb0fc1272350226d3bbe4a21b249241023df481689767c2695953e"} err="failed to get container status \"ffa63de3cfdb0fc1272350226d3bbe4a21b249241023df481689767c2695953e\": rpc error: code = NotFound desc = could not find container \"ffa63de3cfdb0fc1272350226d3bbe4a21b249241023df481689767c2695953e\": container with ID starting with ffa63de3cfdb0fc1272350226d3bbe4a21b249241023df481689767c2695953e not found: ID does not exist"
Sep 16 19:28:18.182449 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.182407    2554 scope.go:117] "RemoveContainer" containerID="bb7ecb377045ea5136b5bbe5bcf58984090d548b7f036d4ad2d4802094124f21"
Sep 16 19:28:18.182619 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.182604    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb7ecb377045ea5136b5bbe5bcf58984090d548b7f036d4ad2d4802094124f21"} err="failed to get container status \"bb7ecb377045ea5136b5bbe5bcf58984090d548b7f036d4ad2d4802094124f21\": rpc error: code = NotFound desc = could not find container \"bb7ecb377045ea5136b5bbe5bcf58984090d548b7f036d4ad2d4802094124f21\": container with ID starting with bb7ecb377045ea5136b5bbe5bcf58984090d548b7f036d4ad2d4802094124f21 not found: ID does not exist"
Sep 16 19:28:18.182670 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.182620    2554 scope.go:117] "RemoveContainer" containerID="fca8126d35a89d5fd5b76ffde016e7b27d1dfee8f86a34dad18c02b94ffa20cb"
Sep 16 19:28:18.182709 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.182676    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"]
Sep 16 19:28:18.182881 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.182861    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fca8126d35a89d5fd5b76ffde016e7b27d1dfee8f86a34dad18c02b94ffa20cb"} err="failed to get container status \"fca8126d35a89d5fd5b76ffde016e7b27d1dfee8f86a34dad18c02b94ffa20cb\": rpc error: code = NotFound desc = could not find container \"fca8126d35a89d5fd5b76ffde016e7b27d1dfee8f86a34dad18c02b94ffa20cb\": container with ID starting with fca8126d35a89d5fd5b76ffde016e7b27d1dfee8f86a34dad18c02b94ffa20cb not found: ID does not exist"
Sep 16 19:28:18.182957 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.182883    2554 scope.go:117] "RemoveContainer" containerID="ffa63de3cfdb0fc1272350226d3bbe4a21b249241023df481689767c2695953e"
Sep 16 19:28:18.183170 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.183145    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffa63de3cfdb0fc1272350226d3bbe4a21b249241023df481689767c2695953e"} err="failed to get container status \"ffa63de3cfdb0fc1272350226d3bbe4a21b249241023df481689767c2695953e\": rpc error: code = NotFound desc = could not find container \"ffa63de3cfdb0fc1272350226d3bbe4a21b249241023df481689767c2695953e\": container with ID starting with ffa63de3cfdb0fc1272350226d3bbe4a21b249241023df481689767c2695953e not found: ID does not exist"
Sep 16 19:28:18.192909 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.186857    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-pn4gs"]
Sep 16 19:28:18.226212 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.226182    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a8a04b7-1602-4f52-99c7-376443de5301-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:28:18.226310 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.226228    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j69pd\" (UniqueName: \"kubernetes.io/projected/7a8a04b7-1602-4f52-99c7-376443de5301-kube-api-access-j69pd\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:28:18.226310 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.226244    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a8a04b7-1602-4f52-99c7-376443de5301-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:28:18.226310 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.226257    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a8a04b7-1602-4f52-99c7-376443de5301-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:28:18.226310 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.226269    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a8a04b7-1602-4f52-99c7-376443de5301-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:28:18.442862 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:18.442797    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a8a04b7-1602-4f52-99c7-376443de5301" path="/var/lib/kubelet/pods/7a8a04b7-1602-4f52-99c7-376443de5301/volumes"
Sep 16 19:28:22.214362 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.214330    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l"]
Sep 16 19:28:22.214745 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.214637    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74e70f40-db89-4920-99b0-92a4a77d76a9" containerName="storage-initializer"
Sep 16 19:28:22.214745 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.214650    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="74e70f40-db89-4920-99b0-92a4a77d76a9" containerName="storage-initializer"
Sep 16 19:28:22.214745 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.214663    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74e70f40-db89-4920-99b0-92a4a77d76a9" containerName="kube-rbac-proxy"
Sep 16 19:28:22.214745 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.214669    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="74e70f40-db89-4920-99b0-92a4a77d76a9" containerName="kube-rbac-proxy"
Sep 16 19:28:22.214745 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.214676    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a8a04b7-1602-4f52-99c7-376443de5301" containerName="storage-initializer"
Sep 16 19:28:22.214745 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.214682    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a8a04b7-1602-4f52-99c7-376443de5301" containerName="storage-initializer"
Sep 16 19:28:22.214745 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.214691    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a8a04b7-1602-4f52-99c7-376443de5301" containerName="kube-rbac-proxy"
Sep 16 19:28:22.214745 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.214697    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a8a04b7-1602-4f52-99c7-376443de5301" containerName="kube-rbac-proxy"
Sep 16 19:28:22.214745 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.214703    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74e70f40-db89-4920-99b0-92a4a77d76a9" containerName="kserve-container"
Sep 16 19:28:22.214745 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.214708    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="74e70f40-db89-4920-99b0-92a4a77d76a9" containerName="kserve-container"
Sep 16 19:28:22.214745 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.214735    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a8a04b7-1602-4f52-99c7-376443de5301" containerName="kserve-container"
Sep 16 19:28:22.214745 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.214740    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a8a04b7-1602-4f52-99c7-376443de5301" containerName="kserve-container"
Sep 16 19:28:22.215146 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.214791    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="74e70f40-db89-4920-99b0-92a4a77d76a9" containerName="kserve-container"
Sep 16 19:28:22.215146 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.214800    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="74e70f40-db89-4920-99b0-92a4a77d76a9" containerName="kube-rbac-proxy"
Sep 16 19:28:22.215146 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.214806    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a8a04b7-1602-4f52-99c7-376443de5301" containerName="kserve-container"
Sep 16 19:28:22.215146 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.214815    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a8a04b7-1602-4f52-99c7-376443de5301" containerName="kube-rbac-proxy"
Sep 16 19:28:22.219754 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.219734    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l"
Sep 16 19:28:22.222024 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.222004    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 16 19:28:22.222146 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.222099    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:28:22.222146 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.222123    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:28:22.222260 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.222126    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8gklb\""
Sep 16 19:28:22.222331 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.222313    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:28:22.222393 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.222377    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Sep 16 19:28:22.222455 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.222402    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:28:22.231433 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.231414    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l"]
Sep 16 19:28:22.356105 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.356084    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd33ca30-fad7-4267-bfc1-5d91384468aa-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l\" (UID: \"dd33ca30-fad7-4267-bfc1-5d91384468aa\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l"
Sep 16 19:28:22.356229 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.356128    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd33ca30-fad7-4267-bfc1-5d91384468aa-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l\" (UID: \"dd33ca30-fad7-4267-bfc1-5d91384468aa\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l"
Sep 16 19:28:22.356229 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.356171    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rgs6\" (UniqueName: \"kubernetes.io/projected/dd33ca30-fad7-4267-bfc1-5d91384468aa-kube-api-access-6rgs6\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l\" (UID: \"dd33ca30-fad7-4267-bfc1-5d91384468aa\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l"
Sep 16 19:28:22.356229 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.356195    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd33ca30-fad7-4267-bfc1-5d91384468aa-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l\" (UID: \"dd33ca30-fad7-4267-bfc1-5d91384468aa\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l"
Sep 16 19:28:22.356346 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.356294    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd33ca30-fad7-4267-bfc1-5d91384468aa-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l\" (UID: \"dd33ca30-fad7-4267-bfc1-5d91384468aa\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l"
Sep 16 19:28:22.456608 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.456585    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd33ca30-fad7-4267-bfc1-5d91384468aa-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l\" (UID: \"dd33ca30-fad7-4267-bfc1-5d91384468aa\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l"
Sep 16 19:28:22.456703 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.456618    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd33ca30-fad7-4267-bfc1-5d91384468aa-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l\" (UID: \"dd33ca30-fad7-4267-bfc1-5d91384468aa\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l"
Sep 16 19:28:22.456703 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.456642    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rgs6\" (UniqueName: \"kubernetes.io/projected/dd33ca30-fad7-4267-bfc1-5d91384468aa-kube-api-access-6rgs6\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l\" (UID: \"dd33ca30-fad7-4267-bfc1-5d91384468aa\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l"
Sep 16 19:28:22.456816 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.456754    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd33ca30-fad7-4267-bfc1-5d91384468aa-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l\" (UID: \"dd33ca30-fad7-4267-bfc1-5d91384468aa\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l"
Sep 16 19:28:22.456867 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.456838    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd33ca30-fad7-4267-bfc1-5d91384468aa-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l\" (UID: \"dd33ca30-fad7-4267-bfc1-5d91384468aa\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l"
Sep 16 19:28:22.457065 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.457040    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd33ca30-fad7-4267-bfc1-5d91384468aa-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l\" (UID: \"dd33ca30-fad7-4267-bfc1-5d91384468aa\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l"
Sep 16 19:28:22.457318 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.457297    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd33ca30-fad7-4267-bfc1-5d91384468aa-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l\" (UID: \"dd33ca30-fad7-4267-bfc1-5d91384468aa\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l"
Sep 16 19:28:22.457393 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.457353    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd33ca30-fad7-4267-bfc1-5d91384468aa-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l\" (UID: \"dd33ca30-fad7-4267-bfc1-5d91384468aa\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l"
Sep 16 19:28:22.459045 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.459025    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd33ca30-fad7-4267-bfc1-5d91384468aa-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l\" (UID: \"dd33ca30-fad7-4267-bfc1-5d91384468aa\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l"
Sep 16 19:28:22.465991 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.465945    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rgs6\" (UniqueName: \"kubernetes.io/projected/dd33ca30-fad7-4267-bfc1-5d91384468aa-kube-api-access-6rgs6\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l\" (UID: \"dd33ca30-fad7-4267-bfc1-5d91384468aa\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l"
Sep 16 19:28:22.529912 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.529890    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l"
Sep 16 19:28:22.803250 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:22.803228    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l"]
Sep 16 19:28:22.805457 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:28:22.805434    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd33ca30_fad7_4267_bfc1_5d91384468aa.slice/crio-5088a017b3045401346a42cb2ccb058e28b08f547beda61c3ebfabbca8d6042f WatchSource:0}: Error finding container 5088a017b3045401346a42cb2ccb058e28b08f547beda61c3ebfabbca8d6042f: Status 404 returned error can't find the container with id 5088a017b3045401346a42cb2ccb058e28b08f547beda61c3ebfabbca8d6042f
Sep 16 19:28:23.184301 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:23.184219    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l" event={"ID":"dd33ca30-fad7-4267-bfc1-5d91384468aa","Type":"ContainerStarted","Data":"2287bbf41d80ed14f42c8e098ff803b13be97a6a3a6e64e6cda9553aef3d211a"}
Sep 16 19:28:23.184301 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:23.184272    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l" event={"ID":"dd33ca30-fad7-4267-bfc1-5d91384468aa","Type":"ContainerStarted","Data":"5088a017b3045401346a42cb2ccb058e28b08f547beda61c3ebfabbca8d6042f"}
Sep 16 19:28:27.203439 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:27.203367    2554 generic.go:356] "Generic (PLEG): container finished" podID="dd33ca30-fad7-4267-bfc1-5d91384468aa" containerID="2287bbf41d80ed14f42c8e098ff803b13be97a6a3a6e64e6cda9553aef3d211a" exitCode=0
Sep 16 19:28:27.203832 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:27.203437    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l" event={"ID":"dd33ca30-fad7-4267-bfc1-5d91384468aa","Type":"ContainerDied","Data":"2287bbf41d80ed14f42c8e098ff803b13be97a6a3a6e64e6cda9553aef3d211a"}
Sep 16 19:28:28.210068 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:28.210030    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l" event={"ID":"dd33ca30-fad7-4267-bfc1-5d91384468aa","Type":"ContainerStarted","Data":"7c5ea3e57ab2a47ad45432b284f50f51f03eff1452a86c20deaddc493c8e034c"}
Sep 16 19:28:28.210439 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:28.210100    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l" event={"ID":"dd33ca30-fad7-4267-bfc1-5d91384468aa","Type":"ContainerStarted","Data":"8e5875b32c99aa090fc7e4573d3b9ecc9d9622d7e2f55c4ca858e3b9cb3759c8"}
Sep 16 19:28:28.210439 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:28.210285    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l"
Sep 16 19:28:28.249319 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:28.249250    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l" podStartSLOduration=2.3370299709999998 podStartE2EDuration="6.249235069s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.912205098s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:28:28.247371221 +0000 UTC m=+2618.323140853" watchObservedRunningTime="2026-09-16 19:28:28.249235069 +0000 UTC m=+2618.325004704"
Sep 16 19:28:29.216700 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:29.216656    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l"
Sep 16 19:28:29.217982 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:29.217953    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l" podUID="dd33ca30-fad7-4267-bfc1-5d91384468aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 16 19:28:30.227958 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:30.227917    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l" podUID="dd33ca30-fad7-4267-bfc1-5d91384468aa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Sep 16 19:28:35.232881 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:35.232848    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l"
Sep 16 19:28:35.234018 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:35.233999    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l"
Sep 16 19:28:42.222917 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:42.222883    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l"]
Sep 16 19:28:42.223344 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:42.223121    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l" podUID="dd33ca30-fad7-4267-bfc1-5d91384468aa" containerName="kserve-container" containerID="cri-o://8e5875b32c99aa090fc7e4573d3b9ecc9d9622d7e2f55c4ca858e3b9cb3759c8" gracePeriod=30
Sep 16 19:28:42.223344 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:42.223165    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l" podUID="dd33ca30-fad7-4267-bfc1-5d91384468aa" containerName="kube-rbac-proxy" containerID="cri-o://7c5ea3e57ab2a47ad45432b284f50f51f03eff1452a86c20deaddc493c8e034c" gracePeriod=30
Sep 16 19:28:42.860498 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:42.860475    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l"
Sep 16 19:28:42.914785 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:42.914732    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd33ca30-fad7-4267-bfc1-5d91384468aa-proxy-tls\") pod \"dd33ca30-fad7-4267-bfc1-5d91384468aa\" (UID: \"dd33ca30-fad7-4267-bfc1-5d91384468aa\") "
Sep 16 19:28:42.914886 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:42.914792    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd33ca30-fad7-4267-bfc1-5d91384468aa-kserve-provision-location\") pod \"dd33ca30-fad7-4267-bfc1-5d91384468aa\" (UID: \"dd33ca30-fad7-4267-bfc1-5d91384468aa\") "
Sep 16 19:28:42.914886 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:42.914813    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd33ca30-fad7-4267-bfc1-5d91384468aa-kube-rbac-proxy-sar-config\") pod \"dd33ca30-fad7-4267-bfc1-5d91384468aa\" (UID: \"dd33ca30-fad7-4267-bfc1-5d91384468aa\") "
Sep 16 19:28:42.914886 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:42.914870    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rgs6\" (UniqueName: \"kubernetes.io/projected/dd33ca30-fad7-4267-bfc1-5d91384468aa-kube-api-access-6rgs6\") pod \"dd33ca30-fad7-4267-bfc1-5d91384468aa\" (UID: \"dd33ca30-fad7-4267-bfc1-5d91384468aa\") "
Sep 16 19:28:42.915020 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:42.914919    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd33ca30-fad7-4267-bfc1-5d91384468aa-cabundle-cert\") pod \"dd33ca30-fad7-4267-bfc1-5d91384468aa\" (UID: \"dd33ca30-fad7-4267-bfc1-5d91384468aa\") "
Sep 16 19:28:42.915208 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:42.915153    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd33ca30-fad7-4267-bfc1-5d91384468aa-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dd33ca30-fad7-4267-bfc1-5d91384468aa" (UID: "dd33ca30-fad7-4267-bfc1-5d91384468aa"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:28:42.915292 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:42.915241    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd33ca30-fad7-4267-bfc1-5d91384468aa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dd33ca30-fad7-4267-bfc1-5d91384468aa" (UID: "dd33ca30-fad7-4267-bfc1-5d91384468aa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:28:42.915347 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:42.915302    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd33ca30-fad7-4267-bfc1-5d91384468aa-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dd33ca30-fad7-4267-bfc1-5d91384468aa" (UID: "dd33ca30-fad7-4267-bfc1-5d91384468aa"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:28:42.916710 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:42.916687    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd33ca30-fad7-4267-bfc1-5d91384468aa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dd33ca30-fad7-4267-bfc1-5d91384468aa" (UID: "dd33ca30-fad7-4267-bfc1-5d91384468aa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:28:42.916829 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:42.916813    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd33ca30-fad7-4267-bfc1-5d91384468aa-kube-api-access-6rgs6" (OuterVolumeSpecName: "kube-api-access-6rgs6") pod "dd33ca30-fad7-4267-bfc1-5d91384468aa" (UID: "dd33ca30-fad7-4267-bfc1-5d91384468aa"). InnerVolumeSpecName "kube-api-access-6rgs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:28:43.016475 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:43.016453    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd33ca30-fad7-4267-bfc1-5d91384468aa-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:28:43.016475 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:43.016473    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd33ca30-fad7-4267-bfc1-5d91384468aa-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:28:43.016595 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:43.016483    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd33ca30-fad7-4267-bfc1-5d91384468aa-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:28:43.016595 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:43.016492    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd33ca30-fad7-4267-bfc1-5d91384468aa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:28:43.016595 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:43.016501    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rgs6\" (UniqueName: \"kubernetes.io/projected/dd33ca30-fad7-4267-bfc1-5d91384468aa-kube-api-access-6rgs6\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:28:43.286078 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:43.286054    2554 generic.go:356] "Generic (PLEG): container finished" podID="dd33ca30-fad7-4267-bfc1-5d91384468aa" containerID="7c5ea3e57ab2a47ad45432b284f50f51f03eff1452a86c20deaddc493c8e034c" exitCode=2
Sep 16 19:28:43.286078 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:43.286074    2554 generic.go:356] "Generic (PLEG): container finished" podID="dd33ca30-fad7-4267-bfc1-5d91384468aa" containerID="8e5875b32c99aa090fc7e4573d3b9ecc9d9622d7e2f55c4ca858e3b9cb3759c8" exitCode=0
Sep 16 19:28:43.286427 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:43.286119    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l" event={"ID":"dd33ca30-fad7-4267-bfc1-5d91384468aa","Type":"ContainerDied","Data":"7c5ea3e57ab2a47ad45432b284f50f51f03eff1452a86c20deaddc493c8e034c"}
Sep 16 19:28:43.286427 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:43.286164    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l" event={"ID":"dd33ca30-fad7-4267-bfc1-5d91384468aa","Type":"ContainerDied","Data":"8e5875b32c99aa090fc7e4573d3b9ecc9d9622d7e2f55c4ca858e3b9cb3759c8"}
Sep 16 19:28:43.286427 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:43.286184    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l" event={"ID":"dd33ca30-fad7-4267-bfc1-5d91384468aa","Type":"ContainerDied","Data":"5088a017b3045401346a42cb2ccb058e28b08f547beda61c3ebfabbca8d6042f"}
Sep 16 19:28:43.286427 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:43.286206    2554 scope.go:117] "RemoveContainer" containerID="7c5ea3e57ab2a47ad45432b284f50f51f03eff1452a86c20deaddc493c8e034c"
Sep 16 19:28:43.286427 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:43.286119    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l"
Sep 16 19:28:43.298435 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:43.298269    2554 scope.go:117] "RemoveContainer" containerID="8e5875b32c99aa090fc7e4573d3b9ecc9d9622d7e2f55c4ca858e3b9cb3759c8"
Sep 16 19:28:43.306169 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:43.306152    2554 scope.go:117] "RemoveContainer" containerID="2287bbf41d80ed14f42c8e098ff803b13be97a6a3a6e64e6cda9553aef3d211a"
Sep 16 19:28:43.313493 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:43.313473    2554 scope.go:117] "RemoveContainer" containerID="7c5ea3e57ab2a47ad45432b284f50f51f03eff1452a86c20deaddc493c8e034c"
Sep 16 19:28:43.313762 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:28:43.313738    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c5ea3e57ab2a47ad45432b284f50f51f03eff1452a86c20deaddc493c8e034c\": container with ID starting with 7c5ea3e57ab2a47ad45432b284f50f51f03eff1452a86c20deaddc493c8e034c not found: ID does not exist" containerID="7c5ea3e57ab2a47ad45432b284f50f51f03eff1452a86c20deaddc493c8e034c"
Sep 16 19:28:43.313886 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:43.313770    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c5ea3e57ab2a47ad45432b284f50f51f03eff1452a86c20deaddc493c8e034c"} err="failed to get container status \"7c5ea3e57ab2a47ad45432b284f50f51f03eff1452a86c20deaddc493c8e034c\": rpc error: code = NotFound desc = could not find container \"7c5ea3e57ab2a47ad45432b284f50f51f03eff1452a86c20deaddc493c8e034c\": container with ID starting with 7c5ea3e57ab2a47ad45432b284f50f51f03eff1452a86c20deaddc493c8e034c not found: ID does not exist"
Sep 16 19:28:43.313886 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:43.313793    2554 scope.go:117] "RemoveContainer" containerID="8e5875b32c99aa090fc7e4573d3b9ecc9d9622d7e2f55c4ca858e3b9cb3759c8"
Sep 16 19:28:43.313989 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:43.313941    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l"]
Sep 16 19:28:43.314048 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:28:43.314034    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e5875b32c99aa090fc7e4573d3b9ecc9d9622d7e2f55c4ca858e3b9cb3759c8\": container with ID starting with 8e5875b32c99aa090fc7e4573d3b9ecc9d9622d7e2f55c4ca858e3b9cb3759c8 not found: ID does not exist" containerID="8e5875b32c99aa090fc7e4573d3b9ecc9d9622d7e2f55c4ca858e3b9cb3759c8"
Sep 16 19:28:43.314089 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:43.314052    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e5875b32c99aa090fc7e4573d3b9ecc9d9622d7e2f55c4ca858e3b9cb3759c8"} err="failed to get container status \"8e5875b32c99aa090fc7e4573d3b9ecc9d9622d7e2f55c4ca858e3b9cb3759c8\": rpc error: code = NotFound desc = could not find container \"8e5875b32c99aa090fc7e4573d3b9ecc9d9622d7e2f55c4ca858e3b9cb3759c8\": container with ID starting with 8e5875b32c99aa090fc7e4573d3b9ecc9d9622d7e2f55c4ca858e3b9cb3759c8 not found: ID does not exist"
Sep 16 19:28:43.314089 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:43.314067    2554 scope.go:117] "RemoveContainer" containerID="2287bbf41d80ed14f42c8e098ff803b13be97a6a3a6e64e6cda9553aef3d211a"
Sep 16 19:28:43.314296 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:28:43.314283    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2287bbf41d80ed14f42c8e098ff803b13be97a6a3a6e64e6cda9553aef3d211a\": container with ID starting with 2287bbf41d80ed14f42c8e098ff803b13be97a6a3a6e64e6cda9553aef3d211a not found: ID does not exist" containerID="2287bbf41d80ed14f42c8e098ff803b13be97a6a3a6e64e6cda9553aef3d211a"
Sep 16 19:28:43.314353 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:43.314297    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2287bbf41d80ed14f42c8e098ff803b13be97a6a3a6e64e6cda9553aef3d211a"} err="failed to get container status \"2287bbf41d80ed14f42c8e098ff803b13be97a6a3a6e64e6cda9553aef3d211a\": rpc error: code = NotFound desc = could not find container \"2287bbf41d80ed14f42c8e098ff803b13be97a6a3a6e64e6cda9553aef3d211a\": container with ID starting with 2287bbf41d80ed14f42c8e098ff803b13be97a6a3a6e64e6cda9553aef3d211a not found: ID does not exist"
Sep 16 19:28:43.314353 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:43.314308    2554 scope.go:117] "RemoveContainer" containerID="7c5ea3e57ab2a47ad45432b284f50f51f03eff1452a86c20deaddc493c8e034c"
Sep 16 19:28:43.314495 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:43.314480    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c5ea3e57ab2a47ad45432b284f50f51f03eff1452a86c20deaddc493c8e034c"} err="failed to get container status \"7c5ea3e57ab2a47ad45432b284f50f51f03eff1452a86c20deaddc493c8e034c\": rpc error: code = NotFound desc = could not find container \"7c5ea3e57ab2a47ad45432b284f50f51f03eff1452a86c20deaddc493c8e034c\": container with ID starting with 7c5ea3e57ab2a47ad45432b284f50f51f03eff1452a86c20deaddc493c8e034c not found: ID does not exist"
Sep 16 19:28:43.314537 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:43.314494    2554 scope.go:117] "RemoveContainer" containerID="8e5875b32c99aa090fc7e4573d3b9ecc9d9622d7e2f55c4ca858e3b9cb3759c8"
Sep 16 19:28:43.314714 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:43.314693    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e5875b32c99aa090fc7e4573d3b9ecc9d9622d7e2f55c4ca858e3b9cb3759c8"} err="failed to get container status \"8e5875b32c99aa090fc7e4573d3b9ecc9d9622d7e2f55c4ca858e3b9cb3759c8\": rpc error: code = NotFound desc = could not find container \"8e5875b32c99aa090fc7e4573d3b9ecc9d9622d7e2f55c4ca858e3b9cb3759c8\": container with ID starting with 8e5875b32c99aa090fc7e4573d3b9ecc9d9622d7e2f55c4ca858e3b9cb3759c8 not found: ID does not exist"
Sep 16 19:28:43.314795 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:43.314737    2554 scope.go:117] "RemoveContainer" containerID="2287bbf41d80ed14f42c8e098ff803b13be97a6a3a6e64e6cda9553aef3d211a"
Sep 16 19:28:43.314996 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:43.314976    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2287bbf41d80ed14f42c8e098ff803b13be97a6a3a6e64e6cda9553aef3d211a"} err="failed to get container status \"2287bbf41d80ed14f42c8e098ff803b13be97a6a3a6e64e6cda9553aef3d211a\": rpc error: code = NotFound desc = could not find container \"2287bbf41d80ed14f42c8e098ff803b13be97a6a3a6e64e6cda9553aef3d211a\": container with ID starting with 2287bbf41d80ed14f42c8e098ff803b13be97a6a3a6e64e6cda9553aef3d211a not found: ID does not exist"
Sep 16 19:28:43.318419 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:43.318399    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-wpr9l"]
Sep 16 19:28:44.442631 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:44.442603    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd33ca30-fad7-4267-bfc1-5d91384468aa" path="/var/lib/kubelet/pods/dd33ca30-fad7-4267-bfc1-5d91384468aa/volumes"
Sep 16 19:28:47.464090 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.464057    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"]
Sep 16 19:28:47.464474 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.464388    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd33ca30-fad7-4267-bfc1-5d91384468aa" containerName="storage-initializer"
Sep 16 19:28:47.464474 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.464399    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd33ca30-fad7-4267-bfc1-5d91384468aa" containerName="storage-initializer"
Sep 16 19:28:47.464474 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.464413    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd33ca30-fad7-4267-bfc1-5d91384468aa" containerName="kserve-container"
Sep 16 19:28:47.464474 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.464418    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd33ca30-fad7-4267-bfc1-5d91384468aa" containerName="kserve-container"
Sep 16 19:28:47.464474 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.464425    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd33ca30-fad7-4267-bfc1-5d91384468aa" containerName="kube-rbac-proxy"
Sep 16 19:28:47.464474 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.464431    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd33ca30-fad7-4267-bfc1-5d91384468aa" containerName="kube-rbac-proxy"
Sep 16 19:28:47.464668 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.464483    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd33ca30-fad7-4267-bfc1-5d91384468aa" containerName="kserve-container"
Sep 16 19:28:47.464668 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.464494    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd33ca30-fad7-4267-bfc1-5d91384468aa" containerName="kube-rbac-proxy"
Sep 16 19:28:47.469265 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.469246    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"
Sep 16 19:28:47.472255 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.472237    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Sep 16 19:28:47.472382 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.472356    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8gklb\""
Sep 16 19:28:47.472637 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.472622    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:28:47.472637 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.472632    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:28:47.472778 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.472750    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:28:47.473377 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.473362    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Sep 16 19:28:47.473429 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.473389    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:28:47.485928 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.485908    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"]
Sep 16 19:28:47.550229 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.550209    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ae7ba3c-382d-494d-8576-c0d768fa1760-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh\" (UID: \"4ae7ba3c-382d-494d-8576-c0d768fa1760\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"
Sep 16 19:28:47.550330 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.550246    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ae7ba3c-382d-494d-8576-c0d768fa1760-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh\" (UID: \"4ae7ba3c-382d-494d-8576-c0d768fa1760\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"
Sep 16 19:28:47.550330 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.550275    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ae7ba3c-382d-494d-8576-c0d768fa1760-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh\" (UID: \"4ae7ba3c-382d-494d-8576-c0d768fa1760\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"
Sep 16 19:28:47.550402 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.550339    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmzc9\" (UniqueName: \"kubernetes.io/projected/4ae7ba3c-382d-494d-8576-c0d768fa1760-kube-api-access-cmzc9\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh\" (UID: \"4ae7ba3c-382d-494d-8576-c0d768fa1760\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"
Sep 16 19:28:47.550437 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.550404    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ae7ba3c-382d-494d-8576-c0d768fa1760-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh\" (UID: \"4ae7ba3c-382d-494d-8576-c0d768fa1760\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"
Sep 16 19:28:47.651004 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.650979    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmzc9\" (UniqueName: \"kubernetes.io/projected/4ae7ba3c-382d-494d-8576-c0d768fa1760-kube-api-access-cmzc9\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh\" (UID: \"4ae7ba3c-382d-494d-8576-c0d768fa1760\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"
Sep 16 19:28:47.651109 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.651034    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ae7ba3c-382d-494d-8576-c0d768fa1760-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh\" (UID: \"4ae7ba3c-382d-494d-8576-c0d768fa1760\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"
Sep 16 19:28:47.651109 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.651061    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ae7ba3c-382d-494d-8576-c0d768fa1760-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh\" (UID: \"4ae7ba3c-382d-494d-8576-c0d768fa1760\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"
Sep 16 19:28:47.651109 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.651087    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ae7ba3c-382d-494d-8576-c0d768fa1760-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh\" (UID: \"4ae7ba3c-382d-494d-8576-c0d768fa1760\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"
Sep 16 19:28:47.651246 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.651115    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ae7ba3c-382d-494d-8576-c0d768fa1760-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh\" (UID: \"4ae7ba3c-382d-494d-8576-c0d768fa1760\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"
Sep 16 19:28:47.651246 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:28:47.651195    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-serving-cert: secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Sep 16 19:28:47.651333 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:28:47.651258    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ae7ba3c-382d-494d-8576-c0d768fa1760-proxy-tls podName:4ae7ba3c-382d-494d-8576-c0d768fa1760 nodeName:}" failed. No retries permitted until 2026-09-16 19:28:48.151238021 +0000 UTC m=+2638.227007638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4ae7ba3c-382d-494d-8576-c0d768fa1760-proxy-tls") pod "isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh" (UID: "4ae7ba3c-382d-494d-8576-c0d768fa1760") : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Sep 16 19:28:47.651483 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.651461    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ae7ba3c-382d-494d-8576-c0d768fa1760-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh\" (UID: \"4ae7ba3c-382d-494d-8576-c0d768fa1760\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"
Sep 16 19:28:47.651749 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.651709    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ae7ba3c-382d-494d-8576-c0d768fa1760-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh\" (UID: \"4ae7ba3c-382d-494d-8576-c0d768fa1760\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"
Sep 16 19:28:47.651813 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.651795    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ae7ba3c-382d-494d-8576-c0d768fa1760-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh\" (UID: \"4ae7ba3c-382d-494d-8576-c0d768fa1760\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"
Sep 16 19:28:47.659826 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:47.659807    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmzc9\" (UniqueName: \"kubernetes.io/projected/4ae7ba3c-382d-494d-8576-c0d768fa1760-kube-api-access-cmzc9\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh\" (UID: \"4ae7ba3c-382d-494d-8576-c0d768fa1760\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"
Sep 16 19:28:48.154299 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:48.154277    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ae7ba3c-382d-494d-8576-c0d768fa1760-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh\" (UID: \"4ae7ba3c-382d-494d-8576-c0d768fa1760\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"
Sep 16 19:28:48.156612 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:48.156589    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ae7ba3c-382d-494d-8576-c0d768fa1760-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh\" (UID: \"4ae7ba3c-382d-494d-8576-c0d768fa1760\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"
Sep 16 19:28:48.379598 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:48.379557    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"
Sep 16 19:28:48.653239 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:48.653211    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"]
Sep 16 19:28:48.655855 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:28:48.655831    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ae7ba3c_382d_494d_8576_c0d768fa1760.slice/crio-8737cb96a38e7251fd62a5c03680a33354149ec107f9b10712f7b6e2e4b5373b WatchSource:0}: Error finding container 8737cb96a38e7251fd62a5c03680a33354149ec107f9b10712f7b6e2e4b5373b: Status 404 returned error can't find the container with id 8737cb96a38e7251fd62a5c03680a33354149ec107f9b10712f7b6e2e4b5373b
Sep 16 19:28:49.316016 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:49.315982    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh" event={"ID":"4ae7ba3c-382d-494d-8576-c0d768fa1760","Type":"ContainerStarted","Data":"0b54d512ca45203d9b144a4c5f2cb3d3ca4c102257d1d5b4557b36eb17904a7d"}
Sep 16 19:28:49.316217 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:49.316048    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh" event={"ID":"4ae7ba3c-382d-494d-8576-c0d768fa1760","Type":"ContainerStarted","Data":"8737cb96a38e7251fd62a5c03680a33354149ec107f9b10712f7b6e2e4b5373b"}
Sep 16 19:28:53.336553 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:53.336520    2554 generic.go:356] "Generic (PLEG): container finished" podID="4ae7ba3c-382d-494d-8576-c0d768fa1760" containerID="0b54d512ca45203d9b144a4c5f2cb3d3ca4c102257d1d5b4557b36eb17904a7d" exitCode=0
Sep 16 19:28:53.336930 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:53.336596    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh" event={"ID":"4ae7ba3c-382d-494d-8576-c0d768fa1760","Type":"ContainerDied","Data":"0b54d512ca45203d9b144a4c5f2cb3d3ca4c102257d1d5b4557b36eb17904a7d"}
Sep 16 19:28:54.343451 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:54.343417    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh" event={"ID":"4ae7ba3c-382d-494d-8576-c0d768fa1760","Type":"ContainerStarted","Data":"5963bd7965e0e9bb2dd1f3bcb61b1ea175e6e49766a3cbb1753ae9670b0f8b3b"}
Sep 16 19:28:54.343895 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:54.343484    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh" event={"ID":"4ae7ba3c-382d-494d-8576-c0d768fa1760","Type":"ContainerStarted","Data":"5ac757d19dcdf90f64f9dfe1da2beaeb32921c19f1584a30b716fdc48c3a330f"}
Sep 16 19:28:54.343895 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:54.343682    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"
Sep 16 19:28:54.343895 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:54.343753    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"
Sep 16 19:28:54.364544 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:28:54.364490    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh" podStartSLOduration=3.309501879 podStartE2EDuration="7.364470601s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.054968722s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:28:54.363063235 +0000 UTC m=+2644.438832866" watchObservedRunningTime="2026-09-16 19:28:54.364470601 +0000 UTC m=+2644.440240236"
Sep 16 19:29:00.353183 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:00.353153    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"
Sep 16 19:29:30.356521 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:30.356494    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"
Sep 16 19:29:37.505634 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:37.505605    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"]
Sep 16 19:29:37.506032 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:37.505852    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh" podUID="4ae7ba3c-382d-494d-8576-c0d768fa1760" containerName="kserve-container" containerID="cri-o://5ac757d19dcdf90f64f9dfe1da2beaeb32921c19f1584a30b716fdc48c3a330f" gracePeriod=30
Sep 16 19:29:37.506032 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:37.505881    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh" podUID="4ae7ba3c-382d-494d-8576-c0d768fa1760" containerName="kube-rbac-proxy" containerID="cri-o://5963bd7965e0e9bb2dd1f3bcb61b1ea175e6e49766a3cbb1753ae9670b0f8b3b" gracePeriod=30
Sep 16 19:29:38.543657 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.543631    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"
Sep 16 19:29:38.546743 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.546702    2554 generic.go:356] "Generic (PLEG): container finished" podID="4ae7ba3c-382d-494d-8576-c0d768fa1760" containerID="5963bd7965e0e9bb2dd1f3bcb61b1ea175e6e49766a3cbb1753ae9670b0f8b3b" exitCode=2
Sep 16 19:29:38.546743 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.546741    2554 generic.go:356] "Generic (PLEG): container finished" podID="4ae7ba3c-382d-494d-8576-c0d768fa1760" containerID="5ac757d19dcdf90f64f9dfe1da2beaeb32921c19f1584a30b716fdc48c3a330f" exitCode=0
Sep 16 19:29:38.546882 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.546743    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh" event={"ID":"4ae7ba3c-382d-494d-8576-c0d768fa1760","Type":"ContainerDied","Data":"5963bd7965e0e9bb2dd1f3bcb61b1ea175e6e49766a3cbb1753ae9670b0f8b3b"}
Sep 16 19:29:38.547147 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.547044    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh" event={"ID":"4ae7ba3c-382d-494d-8576-c0d768fa1760","Type":"ContainerDied","Data":"5ac757d19dcdf90f64f9dfe1da2beaeb32921c19f1584a30b716fdc48c3a330f"}
Sep 16 19:29:38.547251 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.547147    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh" event={"ID":"4ae7ba3c-382d-494d-8576-c0d768fa1760","Type":"ContainerDied","Data":"8737cb96a38e7251fd62a5c03680a33354149ec107f9b10712f7b6e2e4b5373b"}
Sep 16 19:29:38.547251 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.547182    2554 scope.go:117] "RemoveContainer" containerID="5963bd7965e0e9bb2dd1f3bcb61b1ea175e6e49766a3cbb1753ae9670b0f8b3b"
Sep 16 19:29:38.547251 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.547243    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"
Sep 16 19:29:38.560399 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.560373    2554 scope.go:117] "RemoveContainer" containerID="5ac757d19dcdf90f64f9dfe1da2beaeb32921c19f1584a30b716fdc48c3a330f"
Sep 16 19:29:38.568353 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.568332    2554 scope.go:117] "RemoveContainer" containerID="0b54d512ca45203d9b144a4c5f2cb3d3ca4c102257d1d5b4557b36eb17904a7d"
Sep 16 19:29:38.576188 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.576170    2554 scope.go:117] "RemoveContainer" containerID="5963bd7965e0e9bb2dd1f3bcb61b1ea175e6e49766a3cbb1753ae9670b0f8b3b"
Sep 16 19:29:38.576455 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:29:38.576438    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5963bd7965e0e9bb2dd1f3bcb61b1ea175e6e49766a3cbb1753ae9670b0f8b3b\": container with ID starting with 5963bd7965e0e9bb2dd1f3bcb61b1ea175e6e49766a3cbb1753ae9670b0f8b3b not found: ID does not exist" containerID="5963bd7965e0e9bb2dd1f3bcb61b1ea175e6e49766a3cbb1753ae9670b0f8b3b"
Sep 16 19:29:38.576520 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.576461    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5963bd7965e0e9bb2dd1f3bcb61b1ea175e6e49766a3cbb1753ae9670b0f8b3b"} err="failed to get container status \"5963bd7965e0e9bb2dd1f3bcb61b1ea175e6e49766a3cbb1753ae9670b0f8b3b\": rpc error: code = NotFound desc = could not find container \"5963bd7965e0e9bb2dd1f3bcb61b1ea175e6e49766a3cbb1753ae9670b0f8b3b\": container with ID starting with 5963bd7965e0e9bb2dd1f3bcb61b1ea175e6e49766a3cbb1753ae9670b0f8b3b not found: ID does not exist"
Sep 16 19:29:38.576520 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.576479    2554 scope.go:117] "RemoveContainer" containerID="5ac757d19dcdf90f64f9dfe1da2beaeb32921c19f1584a30b716fdc48c3a330f"
Sep 16 19:29:38.576738 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:29:38.576704    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ac757d19dcdf90f64f9dfe1da2beaeb32921c19f1584a30b716fdc48c3a330f\": container with ID starting with 5ac757d19dcdf90f64f9dfe1da2beaeb32921c19f1584a30b716fdc48c3a330f not found: ID does not exist" containerID="5ac757d19dcdf90f64f9dfe1da2beaeb32921c19f1584a30b716fdc48c3a330f"
Sep 16 19:29:38.576783 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.576746    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ac757d19dcdf90f64f9dfe1da2beaeb32921c19f1584a30b716fdc48c3a330f"} err="failed to get container status \"5ac757d19dcdf90f64f9dfe1da2beaeb32921c19f1584a30b716fdc48c3a330f\": rpc error: code = NotFound desc = could not find container \"5ac757d19dcdf90f64f9dfe1da2beaeb32921c19f1584a30b716fdc48c3a330f\": container with ID starting with 5ac757d19dcdf90f64f9dfe1da2beaeb32921c19f1584a30b716fdc48c3a330f not found: ID does not exist"
Sep 16 19:29:38.576783 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.576761    2554 scope.go:117] "RemoveContainer" containerID="0b54d512ca45203d9b144a4c5f2cb3d3ca4c102257d1d5b4557b36eb17904a7d"
Sep 16 19:29:38.576986 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:29:38.576970    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b54d512ca45203d9b144a4c5f2cb3d3ca4c102257d1d5b4557b36eb17904a7d\": container with ID starting with 0b54d512ca45203d9b144a4c5f2cb3d3ca4c102257d1d5b4557b36eb17904a7d not found: ID does not exist" containerID="0b54d512ca45203d9b144a4c5f2cb3d3ca4c102257d1d5b4557b36eb17904a7d"
Sep 16 19:29:38.577028 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.576989    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b54d512ca45203d9b144a4c5f2cb3d3ca4c102257d1d5b4557b36eb17904a7d"} err="failed to get container status \"0b54d512ca45203d9b144a4c5f2cb3d3ca4c102257d1d5b4557b36eb17904a7d\": rpc error: code = NotFound desc = could not find container \"0b54d512ca45203d9b144a4c5f2cb3d3ca4c102257d1d5b4557b36eb17904a7d\": container with ID starting with 0b54d512ca45203d9b144a4c5f2cb3d3ca4c102257d1d5b4557b36eb17904a7d not found: ID does not exist"
Sep 16 19:29:38.577028 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.577003    2554 scope.go:117] "RemoveContainer" containerID="5963bd7965e0e9bb2dd1f3bcb61b1ea175e6e49766a3cbb1753ae9670b0f8b3b"
Sep 16 19:29:38.577247 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.577232    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5963bd7965e0e9bb2dd1f3bcb61b1ea175e6e49766a3cbb1753ae9670b0f8b3b"} err="failed to get container status \"5963bd7965e0e9bb2dd1f3bcb61b1ea175e6e49766a3cbb1753ae9670b0f8b3b\": rpc error: code = NotFound desc = could not find container \"5963bd7965e0e9bb2dd1f3bcb61b1ea175e6e49766a3cbb1753ae9670b0f8b3b\": container with ID starting with 5963bd7965e0e9bb2dd1f3bcb61b1ea175e6e49766a3cbb1753ae9670b0f8b3b not found: ID does not exist"
Sep 16 19:29:38.577304 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.577247    2554 scope.go:117] "RemoveContainer" containerID="5ac757d19dcdf90f64f9dfe1da2beaeb32921c19f1584a30b716fdc48c3a330f"
Sep 16 19:29:38.577473 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.577458    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ac757d19dcdf90f64f9dfe1da2beaeb32921c19f1584a30b716fdc48c3a330f"} err="failed to get container status \"5ac757d19dcdf90f64f9dfe1da2beaeb32921c19f1584a30b716fdc48c3a330f\": rpc error: code = NotFound desc = could not find container \"5ac757d19dcdf90f64f9dfe1da2beaeb32921c19f1584a30b716fdc48c3a330f\": container with ID starting with 5ac757d19dcdf90f64f9dfe1da2beaeb32921c19f1584a30b716fdc48c3a330f not found: ID does not exist"
Sep 16 19:29:38.577523 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.577473    2554 scope.go:117] "RemoveContainer" containerID="0b54d512ca45203d9b144a4c5f2cb3d3ca4c102257d1d5b4557b36eb17904a7d"
Sep 16 19:29:38.577792 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.577696    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b54d512ca45203d9b144a4c5f2cb3d3ca4c102257d1d5b4557b36eb17904a7d"} err="failed to get container status \"0b54d512ca45203d9b144a4c5f2cb3d3ca4c102257d1d5b4557b36eb17904a7d\": rpc error: code = NotFound desc = could not find container \"0b54d512ca45203d9b144a4c5f2cb3d3ca4c102257d1d5b4557b36eb17904a7d\": container with ID starting with 0b54d512ca45203d9b144a4c5f2cb3d3ca4c102257d1d5b4557b36eb17904a7d not found: ID does not exist"
Sep 16 19:29:38.627577 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.627531    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmzc9\" (UniqueName: \"kubernetes.io/projected/4ae7ba3c-382d-494d-8576-c0d768fa1760-kube-api-access-cmzc9\") pod \"4ae7ba3c-382d-494d-8576-c0d768fa1760\" (UID: \"4ae7ba3c-382d-494d-8576-c0d768fa1760\") "
Sep 16 19:29:38.627668 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.627594    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ae7ba3c-382d-494d-8576-c0d768fa1760-cabundle-cert\") pod \"4ae7ba3c-382d-494d-8576-c0d768fa1760\" (UID: \"4ae7ba3c-382d-494d-8576-c0d768fa1760\") "
Sep 16 19:29:38.627711 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.627685    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ae7ba3c-382d-494d-8576-c0d768fa1760-kube-rbac-proxy-sar-config\") pod \"4ae7ba3c-382d-494d-8576-c0d768fa1760\" (UID: \"4ae7ba3c-382d-494d-8576-c0d768fa1760\") "
Sep 16 19:29:38.627788 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.627769    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ae7ba3c-382d-494d-8576-c0d768fa1760-kserve-provision-location\") pod \"4ae7ba3c-382d-494d-8576-c0d768fa1760\" (UID: \"4ae7ba3c-382d-494d-8576-c0d768fa1760\") "
Sep 16 19:29:38.627843 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.627812    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ae7ba3c-382d-494d-8576-c0d768fa1760-proxy-tls\") pod \"4ae7ba3c-382d-494d-8576-c0d768fa1760\" (UID: \"4ae7ba3c-382d-494d-8576-c0d768fa1760\") "
Sep 16 19:29:38.628006 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.627920    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ae7ba3c-382d-494d-8576-c0d768fa1760-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4ae7ba3c-382d-494d-8576-c0d768fa1760" (UID: "4ae7ba3c-382d-494d-8576-c0d768fa1760"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:29:38.628123 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.628073    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ae7ba3c-382d-494d-8576-c0d768fa1760-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4ae7ba3c-382d-494d-8576-c0d768fa1760" (UID: "4ae7ba3c-382d-494d-8576-c0d768fa1760"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:29:38.628123 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.628084    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4ae7ba3c-382d-494d-8576-c0d768fa1760-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:29:38.628123 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.628086    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ae7ba3c-382d-494d-8576-c0d768fa1760-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4ae7ba3c-382d-494d-8576-c0d768fa1760" (UID: "4ae7ba3c-382d-494d-8576-c0d768fa1760"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:29:38.629348 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.629330    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ae7ba3c-382d-494d-8576-c0d768fa1760-kube-api-access-cmzc9" (OuterVolumeSpecName: "kube-api-access-cmzc9") pod "4ae7ba3c-382d-494d-8576-c0d768fa1760" (UID: "4ae7ba3c-382d-494d-8576-c0d768fa1760"). InnerVolumeSpecName "kube-api-access-cmzc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:29:38.629671 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.629656    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ae7ba3c-382d-494d-8576-c0d768fa1760-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4ae7ba3c-382d-494d-8576-c0d768fa1760" (UID: "4ae7ba3c-382d-494d-8576-c0d768fa1760"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:29:38.728881 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.728861    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4ae7ba3c-382d-494d-8576-c0d768fa1760-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:29:38.728881 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.728881    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ae7ba3c-382d-494d-8576-c0d768fa1760-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:29:38.729007 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.728891    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmzc9\" (UniqueName: \"kubernetes.io/projected/4ae7ba3c-382d-494d-8576-c0d768fa1760-kube-api-access-cmzc9\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:29:38.729007 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.728900    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ae7ba3c-382d-494d-8576-c0d768fa1760-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:29:38.868898 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.868870    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"]
Sep 16 19:29:38.874841 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:38.874820    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-fqlrh"]
Sep 16 19:29:40.444314 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:40.444281    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ae7ba3c-382d-494d-8576-c0d768fa1760" path="/var/lib/kubelet/pods/4ae7ba3c-382d-494d-8576-c0d768fa1760/volumes"
Sep 16 19:29:42.692857 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.692809    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"]
Sep 16 19:29:42.693285 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.693169    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ae7ba3c-382d-494d-8576-c0d768fa1760" containerName="storage-initializer"
Sep 16 19:29:42.693285 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.693180    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ae7ba3c-382d-494d-8576-c0d768fa1760" containerName="storage-initializer"
Sep 16 19:29:42.693285 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.693195    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ae7ba3c-382d-494d-8576-c0d768fa1760" containerName="kube-rbac-proxy"
Sep 16 19:29:42.693285 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.693201    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ae7ba3c-382d-494d-8576-c0d768fa1760" containerName="kube-rbac-proxy"
Sep 16 19:29:42.693285 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.693208    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ae7ba3c-382d-494d-8576-c0d768fa1760" containerName="kserve-container"
Sep 16 19:29:42.693285 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.693215    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ae7ba3c-382d-494d-8576-c0d768fa1760" containerName="kserve-container"
Sep 16 19:29:42.693285 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.693264    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ae7ba3c-382d-494d-8576-c0d768fa1760" containerName="kserve-container"
Sep 16 19:29:42.693285 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.693274    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ae7ba3c-382d-494d-8576-c0d768fa1760" containerName="kube-rbac-proxy"
Sep 16 19:29:42.698243 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.698220    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"
Sep 16 19:29:42.700235 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.700211    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Sep 16 19:29:42.700354 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.700215    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:29:42.700505 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.700483    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:29:42.700616 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.700568    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:29:42.700680 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.700631    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Sep 16 19:29:42.700749 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.700710    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:29:42.700930 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.700911    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8gklb\""
Sep 16 19:29:42.712673 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.712649    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"]
Sep 16 19:29:42.764056 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.764022    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e8c499a-4a60-47bb-9659-db9b5bcc2779-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h\" (UID: \"8e8c499a-4a60-47bb-9659-db9b5bcc2779\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"
Sep 16 19:29:42.764240 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.764082    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbkgv\" (UniqueName: \"kubernetes.io/projected/8e8c499a-4a60-47bb-9659-db9b5bcc2779-kube-api-access-cbkgv\") pod \"isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h\" (UID: \"8e8c499a-4a60-47bb-9659-db9b5bcc2779\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"
Sep 16 19:29:42.764240 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.764113    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e8c499a-4a60-47bb-9659-db9b5bcc2779-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h\" (UID: \"8e8c499a-4a60-47bb-9659-db9b5bcc2779\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"
Sep 16 19:29:42.764240 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.764173    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e8c499a-4a60-47bb-9659-db9b5bcc2779-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h\" (UID: \"8e8c499a-4a60-47bb-9659-db9b5bcc2779\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"
Sep 16 19:29:42.764240 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.764225    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e8c499a-4a60-47bb-9659-db9b5bcc2779-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h\" (UID: \"8e8c499a-4a60-47bb-9659-db9b5bcc2779\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"
Sep 16 19:29:42.865057 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.865018    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbkgv\" (UniqueName: \"kubernetes.io/projected/8e8c499a-4a60-47bb-9659-db9b5bcc2779-kube-api-access-cbkgv\") pod \"isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h\" (UID: \"8e8c499a-4a60-47bb-9659-db9b5bcc2779\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"
Sep 16 19:29:42.865258 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.865069    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e8c499a-4a60-47bb-9659-db9b5bcc2779-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h\" (UID: \"8e8c499a-4a60-47bb-9659-db9b5bcc2779\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"
Sep 16 19:29:42.865258 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.865091    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e8c499a-4a60-47bb-9659-db9b5bcc2779-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h\" (UID: \"8e8c499a-4a60-47bb-9659-db9b5bcc2779\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"
Sep 16 19:29:42.865258 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.865132    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e8c499a-4a60-47bb-9659-db9b5bcc2779-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h\" (UID: \"8e8c499a-4a60-47bb-9659-db9b5bcc2779\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"
Sep 16 19:29:42.865258 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.865159    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e8c499a-4a60-47bb-9659-db9b5bcc2779-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h\" (UID: \"8e8c499a-4a60-47bb-9659-db9b5bcc2779\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"
Sep 16 19:29:42.865487 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:29:42.865294    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-serving-cert: secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Sep 16 19:29:42.865487 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:29:42.865370    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e8c499a-4a60-47bb-9659-db9b5bcc2779-proxy-tls podName:8e8c499a-4a60-47bb-9659-db9b5bcc2779 nodeName:}" failed. No retries permitted until 2026-09-16 19:29:43.365355002 +0000 UTC m=+2693.441124625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8e8c499a-4a60-47bb-9659-db9b5bcc2779-proxy-tls") pod "isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h" (UID: "8e8c499a-4a60-47bb-9659-db9b5bcc2779") : secret "isvc-sklearn-mcp-predictor-serving-cert" not found
Sep 16 19:29:42.865614 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.865584    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e8c499a-4a60-47bb-9659-db9b5bcc2779-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h\" (UID: \"8e8c499a-4a60-47bb-9659-db9b5bcc2779\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"
Sep 16 19:29:42.865834 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.865815    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e8c499a-4a60-47bb-9659-db9b5bcc2779-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h\" (UID: \"8e8c499a-4a60-47bb-9659-db9b5bcc2779\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"
Sep 16 19:29:42.865912 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.865845    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e8c499a-4a60-47bb-9659-db9b5bcc2779-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h\" (UID: \"8e8c499a-4a60-47bb-9659-db9b5bcc2779\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"
Sep 16 19:29:42.873435 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:42.873414    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbkgv\" (UniqueName: \"kubernetes.io/projected/8e8c499a-4a60-47bb-9659-db9b5bcc2779-kube-api-access-cbkgv\") pod \"isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h\" (UID: \"8e8c499a-4a60-47bb-9659-db9b5bcc2779\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"
Sep 16 19:29:43.369648 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:43.369617    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e8c499a-4a60-47bb-9659-db9b5bcc2779-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h\" (UID: \"8e8c499a-4a60-47bb-9659-db9b5bcc2779\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"
Sep 16 19:29:43.371980 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:43.371951    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e8c499a-4a60-47bb-9659-db9b5bcc2779-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h\" (UID: \"8e8c499a-4a60-47bb-9659-db9b5bcc2779\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"
Sep 16 19:29:43.608541 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:43.608512    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"
Sep 16 19:29:43.881851 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:43.881828    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"]
Sep 16 19:29:43.883870 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:29:43.883841    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e8c499a_4a60_47bb_9659_db9b5bcc2779.slice/crio-e3fe3f39fbd6ad880ef3660a5f3ca30473ea2ae1bdd1a8f52c6bacdb5fbb7fed WatchSource:0}: Error finding container e3fe3f39fbd6ad880ef3660a5f3ca30473ea2ae1bdd1a8f52c6bacdb5fbb7fed: Status 404 returned error can't find the container with id e3fe3f39fbd6ad880ef3660a5f3ca30473ea2ae1bdd1a8f52c6bacdb5fbb7fed
Sep 16 19:29:44.578563 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:44.578533    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h" event={"ID":"8e8c499a-4a60-47bb-9659-db9b5bcc2779","Type":"ContainerStarted","Data":"ea2339b6be3768ebb56bcd0c78a4d5d922e9ea02cff1410cc6a78bad884242f2"}
Sep 16 19:29:44.578775 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:44.578606    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h" event={"ID":"8e8c499a-4a60-47bb-9659-db9b5bcc2779","Type":"ContainerStarted","Data":"e3fe3f39fbd6ad880ef3660a5f3ca30473ea2ae1bdd1a8f52c6bacdb5fbb7fed"}
Sep 16 19:29:48.598940 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:48.598904    2554 generic.go:356] "Generic (PLEG): container finished" podID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" containerID="ea2339b6be3768ebb56bcd0c78a4d5d922e9ea02cff1410cc6a78bad884242f2" exitCode=0
Sep 16 19:29:48.599344 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:48.598948    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h" event={"ID":"8e8c499a-4a60-47bb-9659-db9b5bcc2779","Type":"ContainerDied","Data":"ea2339b6be3768ebb56bcd0c78a4d5d922e9ea02cff1410cc6a78bad884242f2"}
Sep 16 19:29:49.610205 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:49.610164    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h" event={"ID":"8e8c499a-4a60-47bb-9659-db9b5bcc2779","Type":"ContainerStarted","Data":"584f11729b131251bbe75518017e1302c04a95b67b16a51cdeafe8fa7f1c6c0f"}
Sep 16 19:29:51.434576 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:51.434548    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/2.log"
Sep 16 19:29:51.435047 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:51.434548    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/2.log"
Sep 16 19:29:51.440655 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:51.440633    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovn-acl-logging/0.log"
Sep 16 19:29:51.440810 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:51.440791    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovn-acl-logging/0.log"
Sep 16 19:29:51.622650 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:51.622613    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h" event={"ID":"8e8c499a-4a60-47bb-9659-db9b5bcc2779","Type":"ContainerStarted","Data":"86dc4016fa7504e841b19cf26079974a0232f288925edab30a24bd4acc3f935f"}
Sep 16 19:29:51.622859 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:51.622672    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h" event={"ID":"8e8c499a-4a60-47bb-9659-db9b5bcc2779","Type":"ContainerStarted","Data":"764a2c056381c12db58d794652ceebb0712ba028fd9fcb58e64be024bd2aea58"}
Sep 16 19:29:51.622859 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:51.622701    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"
Sep 16 19:29:51.622859 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:51.622749    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"
Sep 16 19:29:51.646799 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:51.646623    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h" podStartSLOduration=2.9064771719999998 podStartE2EDuration="9.646603207s" totalImagesPullingTime="2.790831227s" totalInitContainerRuntime="3.949294808s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:29:51.644598869 +0000 UTC m=+2701.720368500" watchObservedRunningTime="2026-09-16 19:29:51.646603207 +0000 UTC m=+2701.722372840"
Sep 16 19:29:52.627149 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:52.627121    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"
Sep 16 19:29:58.636940 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:29:58.636911    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"
Sep 16 19:30:18.638699 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:30:18.638662    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h" podUID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.62:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 16 19:30:23.609097 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:30:23.609053    2554 prober.go:120] "Probe failed" probeType="Liveness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h" podUID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.62:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 16 19:30:28.640329 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:30:28.640303    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"
Sep 16 19:30:58.641186 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:30:58.641158    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"
Sep 16 19:31:02.766270 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:02.766231    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"]
Sep 16 19:31:02.766782 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:02.766489    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h" podUID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" containerName="kserve-container" containerID="cri-o://584f11729b131251bbe75518017e1302c04a95b67b16a51cdeafe8fa7f1c6c0f" gracePeriod=30
Sep 16 19:31:02.766782 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:02.766530    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h" podUID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" containerName="kube-rbac-proxy" containerID="cri-o://86dc4016fa7504e841b19cf26079974a0232f288925edab30a24bd4acc3f935f" gracePeriod=30
Sep 16 19:31:02.766782 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:02.766548    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h" podUID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" containerName="kserve-agent" containerID="cri-o://764a2c056381c12db58d794652ceebb0712ba028fd9fcb58e64be024bd2aea58" gracePeriod=30
Sep 16 19:31:02.945326 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:02.945298    2554 generic.go:356] "Generic (PLEG): container finished" podID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" containerID="86dc4016fa7504e841b19cf26079974a0232f288925edab30a24bd4acc3f935f" exitCode=2
Sep 16 19:31:02.945436 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:02.945370    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h" event={"ID":"8e8c499a-4a60-47bb-9659-db9b5bcc2779","Type":"ContainerDied","Data":"86dc4016fa7504e841b19cf26079974a0232f288925edab30a24bd4acc3f935f"}
Sep 16 19:31:03.632666 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:03.632620    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h" podUID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.62:8643/healthz\": dial tcp 10.134.0.62:8643: connect: connection refused"
Sep 16 19:31:04.955805 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:04.955777    2554 generic.go:356] "Generic (PLEG): container finished" podID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" containerID="584f11729b131251bbe75518017e1302c04a95b67b16a51cdeafe8fa7f1c6c0f" exitCode=0
Sep 16 19:31:04.956222 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:04.955848    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h" event={"ID":"8e8c499a-4a60-47bb-9659-db9b5bcc2779","Type":"ContainerDied","Data":"584f11729b131251bbe75518017e1302c04a95b67b16a51cdeafe8fa7f1c6c0f"}
Sep 16 19:31:07.942895 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:07.942865    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv"]
Sep 16 19:31:07.946592 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:07.946573    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv"
Sep 16 19:31:07.948527 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:07.948507    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Sep 16 19:31:07.948640 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:07.948509    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Sep 16 19:31:07.956475 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:07.956455    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv"]
Sep 16 19:31:08.121420 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:08.121394    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/545bba39-549f-4394-a079-60dea15f4947-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nb2mv\" (UID: \"545bba39-549f-4394-a079-60dea15f4947\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv"
Sep 16 19:31:08.121529 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:08.121435    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/545bba39-549f-4394-a079-60dea15f4947-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nb2mv\" (UID: \"545bba39-549f-4394-a079-60dea15f4947\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv"
Sep 16 19:31:08.121529 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:08.121454    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfvck\" (UniqueName: \"kubernetes.io/projected/545bba39-549f-4394-a079-60dea15f4947-kube-api-access-pfvck\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nb2mv\" (UID: \"545bba39-549f-4394-a079-60dea15f4947\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv"
Sep 16 19:31:08.121604 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:08.121523    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/545bba39-549f-4394-a079-60dea15f4947-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nb2mv\" (UID: \"545bba39-549f-4394-a079-60dea15f4947\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv"
Sep 16 19:31:08.121604 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:08.121577    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/545bba39-549f-4394-a079-60dea15f4947-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nb2mv\" (UID: \"545bba39-549f-4394-a079-60dea15f4947\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv"
Sep 16 19:31:08.222261 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:08.222208    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/545bba39-549f-4394-a079-60dea15f4947-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nb2mv\" (UID: \"545bba39-549f-4394-a079-60dea15f4947\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv"
Sep 16 19:31:08.222261 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:08.222251    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/545bba39-549f-4394-a079-60dea15f4947-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nb2mv\" (UID: \"545bba39-549f-4394-a079-60dea15f4947\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv"
Sep 16 19:31:08.222399 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:08.222276    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfvck\" (UniqueName: \"kubernetes.io/projected/545bba39-549f-4394-a079-60dea15f4947-kube-api-access-pfvck\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nb2mv\" (UID: \"545bba39-549f-4394-a079-60dea15f4947\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv"
Sep 16 19:31:08.222399 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:08.222320    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/545bba39-549f-4394-a079-60dea15f4947-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nb2mv\" (UID: \"545bba39-549f-4394-a079-60dea15f4947\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv"
Sep 16 19:31:08.222399 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:08.222366    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/545bba39-549f-4394-a079-60dea15f4947-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nb2mv\" (UID: \"545bba39-549f-4394-a079-60dea15f4947\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv"
Sep 16 19:31:08.222655 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:08.222634    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/545bba39-549f-4394-a079-60dea15f4947-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nb2mv\" (UID: \"545bba39-549f-4394-a079-60dea15f4947\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv"
Sep 16 19:31:08.222930 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:08.222912    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/545bba39-549f-4394-a079-60dea15f4947-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nb2mv\" (UID: \"545bba39-549f-4394-a079-60dea15f4947\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv"
Sep 16 19:31:08.223011 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:08.222927    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/545bba39-549f-4394-a079-60dea15f4947-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nb2mv\" (UID: \"545bba39-549f-4394-a079-60dea15f4947\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv"
Sep 16 19:31:08.224808 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:08.224790    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/545bba39-549f-4394-a079-60dea15f4947-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nb2mv\" (UID: \"545bba39-549f-4394-a079-60dea15f4947\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv"
Sep 16 19:31:08.229973 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:08.229951    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfvck\" (UniqueName: \"kubernetes.io/projected/545bba39-549f-4394-a079-60dea15f4947-kube-api-access-pfvck\") pod \"isvc-paddle-predictor-6d5c9fd4f6-nb2mv\" (UID: \"545bba39-549f-4394-a079-60dea15f4947\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv"
Sep 16 19:31:08.256924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:08.256902    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv"
Sep 16 19:31:08.527687 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:08.527663    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv"]
Sep 16 19:31:08.530097 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:31:08.530073    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod545bba39_549f_4394_a079_60dea15f4947.slice/crio-fc3c680296f5f27b8643e7a8953bb1efb697717f600b0ec87783dcd7d8a24ef5 WatchSource:0}: Error finding container fc3c680296f5f27b8643e7a8953bb1efb697717f600b0ec87783dcd7d8a24ef5: Status 404 returned error can't find the container with id fc3c680296f5f27b8643e7a8953bb1efb697717f600b0ec87783dcd7d8a24ef5
Sep 16 19:31:08.631705 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:08.631667    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h" podUID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.62:8643/healthz\": dial tcp 10.134.0.62:8643: connect: connection refused"
Sep 16 19:31:08.638063 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:08.638033    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h" podUID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.62:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 16 19:31:08.976202 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:08.976120    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv" event={"ID":"545bba39-549f-4394-a079-60dea15f4947","Type":"ContainerStarted","Data":"754669fdc856edb2daff97171a595fe4fa400150338df1b0d6541229342d3137"}
Sep 16 19:31:08.976552 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:08.976244    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv" event={"ID":"545bba39-549f-4394-a079-60dea15f4947","Type":"ContainerStarted","Data":"fc3c680296f5f27b8643e7a8953bb1efb697717f600b0ec87783dcd7d8a24ef5"}
Sep 16 19:31:12.996702 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:12.996673    2554 generic.go:356] "Generic (PLEG): container finished" podID="545bba39-549f-4394-a079-60dea15f4947" containerID="754669fdc856edb2daff97171a595fe4fa400150338df1b0d6541229342d3137" exitCode=0
Sep 16 19:31:12.997007 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:12.996754    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv" event={"ID":"545bba39-549f-4394-a079-60dea15f4947","Type":"ContainerDied","Data":"754669fdc856edb2daff97171a595fe4fa400150338df1b0d6541229342d3137"}
Sep 16 19:31:12.998020 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:12.998003    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:31:13.632796 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:13.632752    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h" podUID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.62:8643/healthz\": dial tcp 10.134.0.62:8643: connect: connection refused"
Sep 16 19:31:13.632987 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:13.632903    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"
Sep 16 19:31:18.631958 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:18.631916    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h" podUID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.62:8643/healthz\": dial tcp 10.134.0.62:8643: connect: connection refused"
Sep 16 19:31:18.638233 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:18.638204    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h" podUID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.62:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 16 19:31:23.632924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:23.632874    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h" podUID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.62:8643/healthz\": dial tcp 10.134.0.62:8643: connect: connection refused"
Sep 16 19:31:25.064306 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:25.064273    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv" event={"ID":"545bba39-549f-4394-a079-60dea15f4947","Type":"ContainerStarted","Data":"1cf475b71f1968abc6da3543c2e5fc6ac8059354b0cb05d236a745610530ef50"}
Sep 16 19:31:25.064767 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:25.064333    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv" event={"ID":"545bba39-549f-4394-a079-60dea15f4947","Type":"ContainerStarted","Data":"00fd59cb98bb085d17ca8e8d586d67f8e72ee3dd855f83437f1ac6685029e878"}
Sep 16 19:31:25.064767 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:25.064596    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv"
Sep 16 19:31:25.064767 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:25.064750    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv"
Sep 16 19:31:25.065791 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:25.065765    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv" podUID="545bba39-549f-4394-a079-60dea15f4947" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 16 19:31:25.087412 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:25.087364    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv" podStartSLOduration=2.06888406 podStartE2EDuration="18.087348436s" totalImagesPullingTime="11.653009384s" totalInitContainerRuntime="4.365454992s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:31:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:31:25.085403479 +0000 UTC m=+2795.161173113" watchObservedRunningTime="2026-09-16 19:31:25.087348436 +0000 UTC m=+2795.163118069"
Sep 16 19:31:26.068784 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:26.068748    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv" podUID="545bba39-549f-4394-a079-60dea15f4947" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 16 19:31:28.632450 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:28.632419    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h" podUID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.62:8643/healthz\": dial tcp 10.134.0.62:8643: connect: connection refused"
Sep 16 19:31:28.637812 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:28.637786    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h" podUID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.62:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.62:8080: connect: connection refused"
Sep 16 19:31:28.637905 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:28.637894    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"
Sep 16 19:31:31.073595 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:31.073566    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv"
Sep 16 19:31:31.074088 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:31.074062    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv" podUID="545bba39-549f-4394-a079-60dea15f4947" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 16 19:31:32.910516 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:32.910494    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"
Sep 16 19:31:33.025954 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.025859    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e8c499a-4a60-47bb-9659-db9b5bcc2779-kserve-provision-location\") pod \"8e8c499a-4a60-47bb-9659-db9b5bcc2779\" (UID: \"8e8c499a-4a60-47bb-9659-db9b5bcc2779\") "
Sep 16 19:31:33.025954 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.025898    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e8c499a-4a60-47bb-9659-db9b5bcc2779-proxy-tls\") pod \"8e8c499a-4a60-47bb-9659-db9b5bcc2779\" (UID: \"8e8c499a-4a60-47bb-9659-db9b5bcc2779\") "
Sep 16 19:31:33.025954 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.025949    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbkgv\" (UniqueName: \"kubernetes.io/projected/8e8c499a-4a60-47bb-9659-db9b5bcc2779-kube-api-access-cbkgv\") pod \"8e8c499a-4a60-47bb-9659-db9b5bcc2779\" (UID: \"8e8c499a-4a60-47bb-9659-db9b5bcc2779\") "
Sep 16 19:31:33.026258 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.025995    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e8c499a-4a60-47bb-9659-db9b5bcc2779-cabundle-cert\") pod \"8e8c499a-4a60-47bb-9659-db9b5bcc2779\" (UID: \"8e8c499a-4a60-47bb-9659-db9b5bcc2779\") "
Sep 16 19:31:33.026258 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.026021    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e8c499a-4a60-47bb-9659-db9b5bcc2779-kube-rbac-proxy-sar-config\") pod \"8e8c499a-4a60-47bb-9659-db9b5bcc2779\" (UID: \"8e8c499a-4a60-47bb-9659-db9b5bcc2779\") "
Sep 16 19:31:33.026374 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.026290    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e8c499a-4a60-47bb-9659-db9b5bcc2779-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8e8c499a-4a60-47bb-9659-db9b5bcc2779" (UID: "8e8c499a-4a60-47bb-9659-db9b5bcc2779"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:31:33.026424 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.026401    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e8c499a-4a60-47bb-9659-db9b5bcc2779-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8e8c499a-4a60-47bb-9659-db9b5bcc2779" (UID: "8e8c499a-4a60-47bb-9659-db9b5bcc2779"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:31:33.026465 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.026443    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e8c499a-4a60-47bb-9659-db9b5bcc2779-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8e8c499a-4a60-47bb-9659-db9b5bcc2779" (UID: "8e8c499a-4a60-47bb-9659-db9b5bcc2779"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:31:33.028154 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.028133    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e8c499a-4a60-47bb-9659-db9b5bcc2779-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8e8c499a-4a60-47bb-9659-db9b5bcc2779" (UID: "8e8c499a-4a60-47bb-9659-db9b5bcc2779"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:31:33.028293 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.028274    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e8c499a-4a60-47bb-9659-db9b5bcc2779-kube-api-access-cbkgv" (OuterVolumeSpecName: "kube-api-access-cbkgv") pod "8e8c499a-4a60-47bb-9659-db9b5bcc2779" (UID: "8e8c499a-4a60-47bb-9659-db9b5bcc2779"). InnerVolumeSpecName "kube-api-access-cbkgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:31:33.103671 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.103630    2554 generic.go:356] "Generic (PLEG): container finished" podID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" containerID="764a2c056381c12db58d794652ceebb0712ba028fd9fcb58e64be024bd2aea58" exitCode=0
Sep 16 19:31:33.103845 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.103708    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"
Sep 16 19:31:33.103845 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.103706    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h" event={"ID":"8e8c499a-4a60-47bb-9659-db9b5bcc2779","Type":"ContainerDied","Data":"764a2c056381c12db58d794652ceebb0712ba028fd9fcb58e64be024bd2aea58"}
Sep 16 19:31:33.103845 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.103786    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h" event={"ID":"8e8c499a-4a60-47bb-9659-db9b5bcc2779","Type":"ContainerDied","Data":"e3fe3f39fbd6ad880ef3660a5f3ca30473ea2ae1bdd1a8f52c6bacdb5fbb7fed"}
Sep 16 19:31:33.103845 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.103817    2554 scope.go:117] "RemoveContainer" containerID="86dc4016fa7504e841b19cf26079974a0232f288925edab30a24bd4acc3f935f"
Sep 16 19:31:33.112928 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.112909    2554 scope.go:117] "RemoveContainer" containerID="764a2c056381c12db58d794652ceebb0712ba028fd9fcb58e64be024bd2aea58"
Sep 16 19:31:33.120082 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.120066    2554 scope.go:117] "RemoveContainer" containerID="584f11729b131251bbe75518017e1302c04a95b67b16a51cdeafe8fa7f1c6c0f"
Sep 16 19:31:33.126296 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.126273    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"]
Sep 16 19:31:33.126710 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.126690    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e8c499a-4a60-47bb-9659-db9b5bcc2779-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:31:33.126782 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.126714    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e8c499a-4a60-47bb-9659-db9b5bcc2779-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:31:33.126782 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.126751    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbkgv\" (UniqueName: \"kubernetes.io/projected/8e8c499a-4a60-47bb-9659-db9b5bcc2779-kube-api-access-cbkgv\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:31:33.126782 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.126760    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e8c499a-4a60-47bb-9659-db9b5bcc2779-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:31:33.126782 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.126768    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e8c499a-4a60-47bb-9659-db9b5bcc2779-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:31:33.128082 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.128060    2554 scope.go:117] "RemoveContainer" containerID="ea2339b6be3768ebb56bcd0c78a4d5d922e9ea02cff1410cc6a78bad884242f2"
Sep 16 19:31:33.132562 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.132539    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-67dbf9ff66-xqt5h"]
Sep 16 19:31:33.135854 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.135839    2554 scope.go:117] "RemoveContainer" containerID="86dc4016fa7504e841b19cf26079974a0232f288925edab30a24bd4acc3f935f"
Sep 16 19:31:33.136119 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:31:33.136100    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86dc4016fa7504e841b19cf26079974a0232f288925edab30a24bd4acc3f935f\": container with ID starting with 86dc4016fa7504e841b19cf26079974a0232f288925edab30a24bd4acc3f935f not found: ID does not exist" containerID="86dc4016fa7504e841b19cf26079974a0232f288925edab30a24bd4acc3f935f"
Sep 16 19:31:33.136169 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.136127    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86dc4016fa7504e841b19cf26079974a0232f288925edab30a24bd4acc3f935f"} err="failed to get container status \"86dc4016fa7504e841b19cf26079974a0232f288925edab30a24bd4acc3f935f\": rpc error: code = NotFound desc = could not find container \"86dc4016fa7504e841b19cf26079974a0232f288925edab30a24bd4acc3f935f\": container with ID starting with 86dc4016fa7504e841b19cf26079974a0232f288925edab30a24bd4acc3f935f not found: ID does not exist"
Sep 16 19:31:33.136169 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.136144    2554 scope.go:117] "RemoveContainer" containerID="764a2c056381c12db58d794652ceebb0712ba028fd9fcb58e64be024bd2aea58"
Sep 16 19:31:33.136394 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:31:33.136376    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"764a2c056381c12db58d794652ceebb0712ba028fd9fcb58e64be024bd2aea58\": container with ID starting with 764a2c056381c12db58d794652ceebb0712ba028fd9fcb58e64be024bd2aea58 not found: ID does not exist" containerID="764a2c056381c12db58d794652ceebb0712ba028fd9fcb58e64be024bd2aea58"
Sep 16 19:31:33.136440 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.136399    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"764a2c056381c12db58d794652ceebb0712ba028fd9fcb58e64be024bd2aea58"} err="failed to get container status \"764a2c056381c12db58d794652ceebb0712ba028fd9fcb58e64be024bd2aea58\": rpc error: code = NotFound desc = could not find container \"764a2c056381c12db58d794652ceebb0712ba028fd9fcb58e64be024bd2aea58\": container with ID starting with 764a2c056381c12db58d794652ceebb0712ba028fd9fcb58e64be024bd2aea58 not found: ID does not exist"
Sep 16 19:31:33.136440 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.136417    2554 scope.go:117] "RemoveContainer" containerID="584f11729b131251bbe75518017e1302c04a95b67b16a51cdeafe8fa7f1c6c0f"
Sep 16 19:31:33.136633 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:31:33.136615    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"584f11729b131251bbe75518017e1302c04a95b67b16a51cdeafe8fa7f1c6c0f\": container with ID starting with 584f11729b131251bbe75518017e1302c04a95b67b16a51cdeafe8fa7f1c6c0f not found: ID does not exist" containerID="584f11729b131251bbe75518017e1302c04a95b67b16a51cdeafe8fa7f1c6c0f"
Sep 16 19:31:33.136699 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.136641    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"584f11729b131251bbe75518017e1302c04a95b67b16a51cdeafe8fa7f1c6c0f"} err="failed to get container status \"584f11729b131251bbe75518017e1302c04a95b67b16a51cdeafe8fa7f1c6c0f\": rpc error: code = NotFound desc = could not find container \"584f11729b131251bbe75518017e1302c04a95b67b16a51cdeafe8fa7f1c6c0f\": container with ID starting with 584f11729b131251bbe75518017e1302c04a95b67b16a51cdeafe8fa7f1c6c0f not found: ID does not exist"
Sep 16 19:31:33.136699 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.136663    2554 scope.go:117] "RemoveContainer" containerID="ea2339b6be3768ebb56bcd0c78a4d5d922e9ea02cff1410cc6a78bad884242f2"
Sep 16 19:31:33.136936 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:31:33.136921    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea2339b6be3768ebb56bcd0c78a4d5d922e9ea02cff1410cc6a78bad884242f2\": container with ID starting with ea2339b6be3768ebb56bcd0c78a4d5d922e9ea02cff1410cc6a78bad884242f2 not found: ID does not exist" containerID="ea2339b6be3768ebb56bcd0c78a4d5d922e9ea02cff1410cc6a78bad884242f2"
Sep 16 19:31:33.136993 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:33.136942    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea2339b6be3768ebb56bcd0c78a4d5d922e9ea02cff1410cc6a78bad884242f2"} err="failed to get container status \"ea2339b6be3768ebb56bcd0c78a4d5d922e9ea02cff1410cc6a78bad884242f2\": rpc error: code = NotFound desc = could not find container \"ea2339b6be3768ebb56bcd0c78a4d5d922e9ea02cff1410cc6a78bad884242f2\": container with ID starting with ea2339b6be3768ebb56bcd0c78a4d5d922e9ea02cff1410cc6a78bad884242f2 not found: ID does not exist"
Sep 16 19:31:34.442859 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:34.442783    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" path="/var/lib/kubelet/pods/8e8c499a-4a60-47bb-9659-db9b5bcc2779/volumes"
Sep 16 19:31:41.074429 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:41.074385    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv" podUID="545bba39-549f-4394-a079-60dea15f4947" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 16 19:31:51.074854 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:31:51.074816    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv" podUID="545bba39-549f-4394-a079-60dea15f4947" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 16 19:32:01.074153 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:01.074110    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv" podUID="545bba39-549f-4394-a079-60dea15f4947" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 16 19:32:07.042978 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.042946    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk"]
Sep 16 19:32:07.043348 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.043320    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" containerName="kserve-agent"
Sep 16 19:32:07.043348 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.043333    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" containerName="kserve-agent"
Sep 16 19:32:07.043348 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.043343    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" containerName="kserve-container"
Sep 16 19:32:07.043348 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.043349    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" containerName="kserve-container"
Sep 16 19:32:07.043484 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.043355    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" containerName="kube-rbac-proxy"
Sep 16 19:32:07.043484 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.043362    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" containerName="kube-rbac-proxy"
Sep 16 19:32:07.043484 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.043383    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" containerName="storage-initializer"
Sep 16 19:32:07.043484 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.043392    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" containerName="storage-initializer"
Sep 16 19:32:07.043484 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.043455    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" containerName="kserve-agent"
Sep 16 19:32:07.043484 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.043467    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" containerName="kserve-container"
Sep 16 19:32:07.043484 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.043479    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e8c499a-4a60-47bb-9659-db9b5bcc2779" containerName="kube-rbac-proxy"
Sep 16 19:32:07.045600 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.045583    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk"
Sep 16 19:32:07.048443 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.048420    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 16 19:32:07.048562 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.048429    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Sep 16 19:32:07.048562 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.048429    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Sep 16 19:32:07.058337 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.058313    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk"]
Sep 16 19:32:07.194919 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.194895    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jn48\" (UniqueName: \"kubernetes.io/projected/f5793b0e-0970-4df1-b61f-6c0812edad60-kube-api-access-2jn48\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk\" (UID: \"f5793b0e-0970-4df1-b61f-6c0812edad60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk"
Sep 16 19:32:07.195043 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.194926    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5793b0e-0970-4df1-b61f-6c0812edad60-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk\" (UID: \"f5793b0e-0970-4df1-b61f-6c0812edad60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk"
Sep 16 19:32:07.195043 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.194955    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5793b0e-0970-4df1-b61f-6c0812edad60-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk\" (UID: \"f5793b0e-0970-4df1-b61f-6c0812edad60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk"
Sep 16 19:32:07.195154 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.195072    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5793b0e-0970-4df1-b61f-6c0812edad60-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk\" (UID: \"f5793b0e-0970-4df1-b61f-6c0812edad60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk"
Sep 16 19:32:07.296424 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.296364    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jn48\" (UniqueName: \"kubernetes.io/projected/f5793b0e-0970-4df1-b61f-6c0812edad60-kube-api-access-2jn48\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk\" (UID: \"f5793b0e-0970-4df1-b61f-6c0812edad60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk"
Sep 16 19:32:07.296424 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.296398    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5793b0e-0970-4df1-b61f-6c0812edad60-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk\" (UID: \"f5793b0e-0970-4df1-b61f-6c0812edad60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk"
Sep 16 19:32:07.296575 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.296426    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5793b0e-0970-4df1-b61f-6c0812edad60-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk\" (UID: \"f5793b0e-0970-4df1-b61f-6c0812edad60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk"
Sep 16 19:32:07.296620 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.296583    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5793b0e-0970-4df1-b61f-6c0812edad60-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk\" (UID: \"f5793b0e-0970-4df1-b61f-6c0812edad60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk"
Sep 16 19:32:07.296944 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.296922    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5793b0e-0970-4df1-b61f-6c0812edad60-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk\" (UID: \"f5793b0e-0970-4df1-b61f-6c0812edad60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk"
Sep 16 19:32:07.297158 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.297143    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5793b0e-0970-4df1-b61f-6c0812edad60-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk\" (UID: \"f5793b0e-0970-4df1-b61f-6c0812edad60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk"
Sep 16 19:32:07.298797 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.298780    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5793b0e-0970-4df1-b61f-6c0812edad60-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk\" (UID: \"f5793b0e-0970-4df1-b61f-6c0812edad60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk"
Sep 16 19:32:07.305688 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.305669    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jn48\" (UniqueName: \"kubernetes.io/projected/f5793b0e-0970-4df1-b61f-6c0812edad60-kube-api-access-2jn48\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk\" (UID: \"f5793b0e-0970-4df1-b61f-6c0812edad60\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk"
Sep 16 19:32:07.355636 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.355614    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk"
Sep 16 19:32:07.626377 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:07.626354    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk"]
Sep 16 19:32:07.628294 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:32:07.628266    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5793b0e_0970_4df1_b61f_6c0812edad60.slice/crio-2a49c6a08b7c636ded39e03e985dd106040c8ed841ae9c3c58ae5d3802604777 WatchSource:0}: Error finding container 2a49c6a08b7c636ded39e03e985dd106040c8ed841ae9c3c58ae5d3802604777: Status 404 returned error can't find the container with id 2a49c6a08b7c636ded39e03e985dd106040c8ed841ae9c3c58ae5d3802604777
Sep 16 19:32:08.269969 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:08.269931    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk" event={"ID":"f5793b0e-0970-4df1-b61f-6c0812edad60","Type":"ContainerStarted","Data":"415aa1b19e9f9cd4453885bb25c939147117cca6c98c9dfc4312d56ae70aaaa5"}
Sep 16 19:32:08.270343 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:08.270000    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk" event={"ID":"f5793b0e-0970-4df1-b61f-6c0812edad60","Type":"ContainerStarted","Data":"2a49c6a08b7c636ded39e03e985dd106040c8ed841ae9c3c58ae5d3802604777"}
Sep 16 19:32:11.075554 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:11.075524    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv"
Sep 16 19:32:14.300080 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:14.300056    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk_f5793b0e-0970-4df1-b61f-6c0812edad60/storage-initializer/0.log"
Sep 16 19:32:14.300393 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:14.300094    2554 generic.go:356] "Generic (PLEG): container finished" podID="f5793b0e-0970-4df1-b61f-6c0812edad60" containerID="415aa1b19e9f9cd4453885bb25c939147117cca6c98c9dfc4312d56ae70aaaa5" exitCode=1
Sep 16 19:32:14.300393 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:14.300156    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk" event={"ID":"f5793b0e-0970-4df1-b61f-6c0812edad60","Type":"ContainerDied","Data":"415aa1b19e9f9cd4453885bb25c939147117cca6c98c9dfc4312d56ae70aaaa5"}
Sep 16 19:32:15.306399 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:15.306370    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk_f5793b0e-0970-4df1-b61f-6c0812edad60/storage-initializer/0.log"
Sep 16 19:32:15.306782 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:15.306460    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk" event={"ID":"f5793b0e-0970-4df1-b61f-6c0812edad60","Type":"ContainerStarted","Data":"a2c095888202b395efcaf0cf410f792300f09d4340018532d528a7fb2a20bea1"}
Sep 16 19:32:17.053634 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:17.053603    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk"]
Sep 16 19:32:17.053994 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:17.053822    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk" podUID="f5793b0e-0970-4df1-b61f-6c0812edad60" containerName="storage-initializer" containerID="cri-o://a2c095888202b395efcaf0cf410f792300f09d4340018532d528a7fb2a20bea1" gracePeriod=30
Sep 16 19:32:19.139098 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:19.139069    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv"]
Sep 16 19:32:19.139476 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:19.139296    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv" podUID="545bba39-549f-4394-a079-60dea15f4947" containerName="kserve-container" containerID="cri-o://00fd59cb98bb085d17ca8e8d586d67f8e72ee3dd855f83437f1ac6685029e878" gracePeriod=30
Sep 16 19:32:19.139476 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:19.139361    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv" podUID="545bba39-549f-4394-a079-60dea15f4947" containerName="kube-rbac-proxy" containerID="cri-o://1cf475b71f1968abc6da3543c2e5fc6ac8059354b0cb05d236a745610530ef50" gracePeriod=30
Sep 16 19:32:19.326572 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:19.326546    2554 generic.go:356] "Generic (PLEG): container finished" podID="545bba39-549f-4394-a079-60dea15f4947" containerID="1cf475b71f1968abc6da3543c2e5fc6ac8059354b0cb05d236a745610530ef50" exitCode=2
Sep 16 19:32:19.326681 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:19.326616    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv" event={"ID":"545bba39-549f-4394-a079-60dea15f4947","Type":"ContainerDied","Data":"1cf475b71f1968abc6da3543c2e5fc6ac8059354b0cb05d236a745610530ef50"}
Sep 16 19:32:19.786674 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:19.786654    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk_f5793b0e-0970-4df1-b61f-6c0812edad60/storage-initializer/1.log"
Sep 16 19:32:19.787100 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:19.787085    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk_f5793b0e-0970-4df1-b61f-6c0812edad60/storage-initializer/0.log"
Sep 16 19:32:19.788429 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:19.788414    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk"
Sep 16 19:32:19.790317 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:19.790302    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5793b0e-0970-4df1-b61f-6c0812edad60-kserve-provision-location\") pod \"f5793b0e-0970-4df1-b61f-6c0812edad60\" (UID: \"f5793b0e-0970-4df1-b61f-6c0812edad60\") "
Sep 16 19:32:19.790381 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:19.790334    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5793b0e-0970-4df1-b61f-6c0812edad60-kube-rbac-proxy-sar-config\") pod \"f5793b0e-0970-4df1-b61f-6c0812edad60\" (UID: \"f5793b0e-0970-4df1-b61f-6c0812edad60\") "
Sep 16 19:32:19.790381 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:19.790358    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5793b0e-0970-4df1-b61f-6c0812edad60-proxy-tls\") pod \"f5793b0e-0970-4df1-b61f-6c0812edad60\" (UID: \"f5793b0e-0970-4df1-b61f-6c0812edad60\") "
Sep 16 19:32:19.790492 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:19.790384    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jn48\" (UniqueName: \"kubernetes.io/projected/f5793b0e-0970-4df1-b61f-6c0812edad60-kube-api-access-2jn48\") pod \"f5793b0e-0970-4df1-b61f-6c0812edad60\" (UID: \"f5793b0e-0970-4df1-b61f-6c0812edad60\") "
Sep 16 19:32:19.790595 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:19.790575    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5793b0e-0970-4df1-b61f-6c0812edad60-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f5793b0e-0970-4df1-b61f-6c0812edad60" (UID: "f5793b0e-0970-4df1-b61f-6c0812edad60"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:19.790708 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:19.790687    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5793b0e-0970-4df1-b61f-6c0812edad60-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f5793b0e-0970-4df1-b61f-6c0812edad60" (UID: "f5793b0e-0970-4df1-b61f-6c0812edad60"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:32:19.792374 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:19.792353    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5793b0e-0970-4df1-b61f-6c0812edad60-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f5793b0e-0970-4df1-b61f-6c0812edad60" (UID: "f5793b0e-0970-4df1-b61f-6c0812edad60"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:32:19.792508 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:19.792485    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5793b0e-0970-4df1-b61f-6c0812edad60-kube-api-access-2jn48" (OuterVolumeSpecName: "kube-api-access-2jn48") pod "f5793b0e-0970-4df1-b61f-6c0812edad60" (UID: "f5793b0e-0970-4df1-b61f-6c0812edad60"). InnerVolumeSpecName "kube-api-access-2jn48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:32:19.891000 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:19.890932    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f5793b0e-0970-4df1-b61f-6c0812edad60-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:32:19.891000 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:19.890965    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f5793b0e-0970-4df1-b61f-6c0812edad60-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:32:19.891000 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:19.890981    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f5793b0e-0970-4df1-b61f-6c0812edad60-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:32:19.891000 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:19.890996    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jn48\" (UniqueName: \"kubernetes.io/projected/f5793b0e-0970-4df1-b61f-6c0812edad60-kube-api-access-2jn48\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:32:20.332838 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:20.332811    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk_f5793b0e-0970-4df1-b61f-6c0812edad60/storage-initializer/1.log"
Sep 16 19:32:20.333256 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:20.333242    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk_f5793b0e-0970-4df1-b61f-6c0812edad60/storage-initializer/0.log"
Sep 16 19:32:20.333325 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:20.333277    2554 generic.go:356] "Generic (PLEG): container finished" podID="f5793b0e-0970-4df1-b61f-6c0812edad60" containerID="a2c095888202b395efcaf0cf410f792300f09d4340018532d528a7fb2a20bea1" exitCode=1
Sep 16 19:32:20.333370 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:20.333336    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk"
Sep 16 19:32:20.333370 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:20.333344    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk" event={"ID":"f5793b0e-0970-4df1-b61f-6c0812edad60","Type":"ContainerDied","Data":"a2c095888202b395efcaf0cf410f792300f09d4340018532d528a7fb2a20bea1"}
Sep 16 19:32:20.333450 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:20.333407    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk" event={"ID":"f5793b0e-0970-4df1-b61f-6c0812edad60","Type":"ContainerDied","Data":"2a49c6a08b7c636ded39e03e985dd106040c8ed841ae9c3c58ae5d3802604777"}
Sep 16 19:32:20.333450 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:20.333444    2554 scope.go:117] "RemoveContainer" containerID="a2c095888202b395efcaf0cf410f792300f09d4340018532d528a7fb2a20bea1"
Sep 16 19:32:20.341884 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:20.341866    2554 scope.go:117] "RemoveContainer" containerID="415aa1b19e9f9cd4453885bb25c939147117cca6c98c9dfc4312d56ae70aaaa5"
Sep 16 19:32:20.349128 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:20.349112    2554 scope.go:117] "RemoveContainer" containerID="a2c095888202b395efcaf0cf410f792300f09d4340018532d528a7fb2a20bea1"
Sep 16 19:32:20.349393 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:32:20.349363    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2c095888202b395efcaf0cf410f792300f09d4340018532d528a7fb2a20bea1\": container with ID starting with a2c095888202b395efcaf0cf410f792300f09d4340018532d528a7fb2a20bea1 not found: ID does not exist" containerID="a2c095888202b395efcaf0cf410f792300f09d4340018532d528a7fb2a20bea1"
Sep 16 19:32:20.349460 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:20.349389    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c095888202b395efcaf0cf410f792300f09d4340018532d528a7fb2a20bea1"} err="failed to get container status \"a2c095888202b395efcaf0cf410f792300f09d4340018532d528a7fb2a20bea1\": rpc error: code = NotFound desc = could not find container \"a2c095888202b395efcaf0cf410f792300f09d4340018532d528a7fb2a20bea1\": container with ID starting with a2c095888202b395efcaf0cf410f792300f09d4340018532d528a7fb2a20bea1 not found: ID does not exist"
Sep 16 19:32:20.349460 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:20.349406    2554 scope.go:117] "RemoveContainer" containerID="415aa1b19e9f9cd4453885bb25c939147117cca6c98c9dfc4312d56ae70aaaa5"
Sep 16 19:32:20.349622 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:32:20.349609    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"415aa1b19e9f9cd4453885bb25c939147117cca6c98c9dfc4312d56ae70aaaa5\": container with ID starting with 415aa1b19e9f9cd4453885bb25c939147117cca6c98c9dfc4312d56ae70aaaa5 not found: ID does not exist" containerID="415aa1b19e9f9cd4453885bb25c939147117cca6c98c9dfc4312d56ae70aaaa5"
Sep 16 19:32:20.349666 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:20.349626    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"415aa1b19e9f9cd4453885bb25c939147117cca6c98c9dfc4312d56ae70aaaa5"} err="failed to get container status \"415aa1b19e9f9cd4453885bb25c939147117cca6c98c9dfc4312d56ae70aaaa5\": rpc error: code = NotFound desc = could not find container \"415aa1b19e9f9cd4453885bb25c939147117cca6c98c9dfc4312d56ae70aaaa5\": container with ID starting with 415aa1b19e9f9cd4453885bb25c939147117cca6c98c9dfc4312d56ae70aaaa5 not found: ID does not exist"
Sep 16 19:32:20.371121 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:20.371093    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk"]
Sep 16 19:32:20.376187 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:20.376164    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5f46d9d44c-j44xk"]
Sep 16 19:32:20.449119 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:20.449090    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5793b0e-0970-4df1-b61f-6c0812edad60" path="/var/lib/kubelet/pods/f5793b0e-0970-4df1-b61f-6c0812edad60/volumes"
Sep 16 19:32:21.069329 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:21.069292    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv" podUID="545bba39-549f-4394-a079-60dea15f4947" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.63:8643/healthz\": dial tcp 10.134.0.63:8643: connect: connection refused"
Sep 16 19:32:21.074180 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:21.074158    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv" podUID="545bba39-549f-4394-a079-60dea15f4947" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.63:8080: connect: connection refused"
Sep 16 19:32:24.683094 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:24.683072    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv"
Sep 16 19:32:24.728994 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:24.728939    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/545bba39-549f-4394-a079-60dea15f4947-cabundle-cert\") pod \"545bba39-549f-4394-a079-60dea15f4947\" (UID: \"545bba39-549f-4394-a079-60dea15f4947\") "
Sep 16 19:32:24.728994 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:24.728987    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/545bba39-549f-4394-a079-60dea15f4947-kserve-provision-location\") pod \"545bba39-549f-4394-a079-60dea15f4947\" (UID: \"545bba39-549f-4394-a079-60dea15f4947\") "
Sep 16 19:32:24.729163 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:24.729009    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfvck\" (UniqueName: \"kubernetes.io/projected/545bba39-549f-4394-a079-60dea15f4947-kube-api-access-pfvck\") pod \"545bba39-549f-4394-a079-60dea15f4947\" (UID: \"545bba39-549f-4394-a079-60dea15f4947\") "
Sep 16 19:32:24.729163 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:24.729056    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/545bba39-549f-4394-a079-60dea15f4947-proxy-tls\") pod \"545bba39-549f-4394-a079-60dea15f4947\" (UID: \"545bba39-549f-4394-a079-60dea15f4947\") "
Sep 16 19:32:24.729163 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:24.729079    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/545bba39-549f-4394-a079-60dea15f4947-kube-rbac-proxy-sar-config\") pod \"545bba39-549f-4394-a079-60dea15f4947\" (UID: \"545bba39-549f-4394-a079-60dea15f4947\") "
Sep 16 19:32:24.729316 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:24.729264    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/545bba39-549f-4394-a079-60dea15f4947-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "545bba39-549f-4394-a079-60dea15f4947" (UID: "545bba39-549f-4394-a079-60dea15f4947"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:32:24.729378 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:24.729353    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/545bba39-549f-4394-a079-60dea15f4947-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:32:24.729518 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:24.729496    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/545bba39-549f-4394-a079-60dea15f4947-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "545bba39-549f-4394-a079-60dea15f4947" (UID: "545bba39-549f-4394-a079-60dea15f4947"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:32:24.731150 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:24.731126    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/545bba39-549f-4394-a079-60dea15f4947-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "545bba39-549f-4394-a079-60dea15f4947" (UID: "545bba39-549f-4394-a079-60dea15f4947"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:32:24.731225 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:24.731188    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/545bba39-549f-4394-a079-60dea15f4947-kube-api-access-pfvck" (OuterVolumeSpecName: "kube-api-access-pfvck") pod "545bba39-549f-4394-a079-60dea15f4947" (UID: "545bba39-549f-4394-a079-60dea15f4947"). InnerVolumeSpecName "kube-api-access-pfvck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:32:24.738542 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:24.738521    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/545bba39-549f-4394-a079-60dea15f4947-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "545bba39-549f-4394-a079-60dea15f4947" (UID: "545bba39-549f-4394-a079-60dea15f4947"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:32:24.830055 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:24.830020    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/545bba39-549f-4394-a079-60dea15f4947-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:32:24.830055 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:24.830051    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfvck\" (UniqueName: \"kubernetes.io/projected/545bba39-549f-4394-a079-60dea15f4947-kube-api-access-pfvck\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:32:24.830055 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:24.830061    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/545bba39-549f-4394-a079-60dea15f4947-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:32:24.830215 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:24.830070    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/545bba39-549f-4394-a079-60dea15f4947-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:32:25.358820 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:25.358785    2554 generic.go:356] "Generic (PLEG): container finished" podID="545bba39-549f-4394-a079-60dea15f4947" containerID="00fd59cb98bb085d17ca8e8d586d67f8e72ee3dd855f83437f1ac6685029e878" exitCode=0
Sep 16 19:32:25.358999 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:25.358852    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv"
Sep 16 19:32:25.358999 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:25.358869    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv" event={"ID":"545bba39-549f-4394-a079-60dea15f4947","Type":"ContainerDied","Data":"00fd59cb98bb085d17ca8e8d586d67f8e72ee3dd855f83437f1ac6685029e878"}
Sep 16 19:32:25.358999 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:25.358929    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv" event={"ID":"545bba39-549f-4394-a079-60dea15f4947","Type":"ContainerDied","Data":"fc3c680296f5f27b8643e7a8953bb1efb697717f600b0ec87783dcd7d8a24ef5"}
Sep 16 19:32:25.358999 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:25.358956    2554 scope.go:117] "RemoveContainer" containerID="1cf475b71f1968abc6da3543c2e5fc6ac8059354b0cb05d236a745610530ef50"
Sep 16 19:32:25.367779 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:25.367757    2554 scope.go:117] "RemoveContainer" containerID="00fd59cb98bb085d17ca8e8d586d67f8e72ee3dd855f83437f1ac6685029e878"
Sep 16 19:32:25.374985 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:25.374968    2554 scope.go:117] "RemoveContainer" containerID="754669fdc856edb2daff97171a595fe4fa400150338df1b0d6541229342d3137"
Sep 16 19:32:25.381266 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:25.381245    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv"]
Sep 16 19:32:25.383291 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:25.383276    2554 scope.go:117] "RemoveContainer" containerID="1cf475b71f1968abc6da3543c2e5fc6ac8059354b0cb05d236a745610530ef50"
Sep 16 19:32:25.383563 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:32:25.383547    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cf475b71f1968abc6da3543c2e5fc6ac8059354b0cb05d236a745610530ef50\": container with ID starting with 1cf475b71f1968abc6da3543c2e5fc6ac8059354b0cb05d236a745610530ef50 not found: ID does not exist" containerID="1cf475b71f1968abc6da3543c2e5fc6ac8059354b0cb05d236a745610530ef50"
Sep 16 19:32:25.383606 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:25.383571    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cf475b71f1968abc6da3543c2e5fc6ac8059354b0cb05d236a745610530ef50"} err="failed to get container status \"1cf475b71f1968abc6da3543c2e5fc6ac8059354b0cb05d236a745610530ef50\": rpc error: code = NotFound desc = could not find container \"1cf475b71f1968abc6da3543c2e5fc6ac8059354b0cb05d236a745610530ef50\": container with ID starting with 1cf475b71f1968abc6da3543c2e5fc6ac8059354b0cb05d236a745610530ef50 not found: ID does not exist"
Sep 16 19:32:25.383606 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:25.383586    2554 scope.go:117] "RemoveContainer" containerID="00fd59cb98bb085d17ca8e8d586d67f8e72ee3dd855f83437f1ac6685029e878"
Sep 16 19:32:25.383850 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:32:25.383835    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00fd59cb98bb085d17ca8e8d586d67f8e72ee3dd855f83437f1ac6685029e878\": container with ID starting with 00fd59cb98bb085d17ca8e8d586d67f8e72ee3dd855f83437f1ac6685029e878 not found: ID does not exist" containerID="00fd59cb98bb085d17ca8e8d586d67f8e72ee3dd855f83437f1ac6685029e878"
Sep 16 19:32:25.383894 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:25.383855    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00fd59cb98bb085d17ca8e8d586d67f8e72ee3dd855f83437f1ac6685029e878"} err="failed to get container status \"00fd59cb98bb085d17ca8e8d586d67f8e72ee3dd855f83437f1ac6685029e878\": rpc error: code = NotFound desc = could not find container \"00fd59cb98bb085d17ca8e8d586d67f8e72ee3dd855f83437f1ac6685029e878\": container with ID starting with 00fd59cb98bb085d17ca8e8d586d67f8e72ee3dd855f83437f1ac6685029e878 not found: ID does not exist"
Sep 16 19:32:25.383894 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:25.383870    2554 scope.go:117] "RemoveContainer" containerID="754669fdc856edb2daff97171a595fe4fa400150338df1b0d6541229342d3137"
Sep 16 19:32:25.384128 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:32:25.384111    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"754669fdc856edb2daff97171a595fe4fa400150338df1b0d6541229342d3137\": container with ID starting with 754669fdc856edb2daff97171a595fe4fa400150338df1b0d6541229342d3137 not found: ID does not exist" containerID="754669fdc856edb2daff97171a595fe4fa400150338df1b0d6541229342d3137"
Sep 16 19:32:25.384178 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:25.384134    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"754669fdc856edb2daff97171a595fe4fa400150338df1b0d6541229342d3137"} err="failed to get container status \"754669fdc856edb2daff97171a595fe4fa400150338df1b0d6541229342d3137\": rpc error: code = NotFound desc = could not find container \"754669fdc856edb2daff97171a595fe4fa400150338df1b0d6541229342d3137\": container with ID starting with 754669fdc856edb2daff97171a595fe4fa400150338df1b0d6541229342d3137 not found: ID does not exist"
Sep 16 19:32:25.384220 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:25.384194    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-nb2mv"]
Sep 16 19:32:26.443276 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:32:26.443245    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="545bba39-549f-4394-a079-60dea15f4947" path="/var/lib/kubelet/pods/545bba39-549f-4394-a079-60dea15f4947/volumes"
Sep 16 19:33:40.546214 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.546183    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"]
Sep 16 19:33:40.546601 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.546532    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5793b0e-0970-4df1-b61f-6c0812edad60" containerName="storage-initializer"
Sep 16 19:33:40.546601 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.546544    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5793b0e-0970-4df1-b61f-6c0812edad60" containerName="storage-initializer"
Sep 16 19:33:40.546601 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.546554    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="545bba39-549f-4394-a079-60dea15f4947" containerName="kserve-container"
Sep 16 19:33:40.546601 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.546559    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="545bba39-549f-4394-a079-60dea15f4947" containerName="kserve-container"
Sep 16 19:33:40.546601 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.546570    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="545bba39-549f-4394-a079-60dea15f4947" containerName="kube-rbac-proxy"
Sep 16 19:33:40.546601 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.546576    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="545bba39-549f-4394-a079-60dea15f4947" containerName="kube-rbac-proxy"
Sep 16 19:33:40.546601 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.546587    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="545bba39-549f-4394-a079-60dea15f4947" containerName="storage-initializer"
Sep 16 19:33:40.546601 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.546595    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="545bba39-549f-4394-a079-60dea15f4947" containerName="storage-initializer"
Sep 16 19:33:40.546868 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.546654    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5793b0e-0970-4df1-b61f-6c0812edad60" containerName="storage-initializer"
Sep 16 19:33:40.546868 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.546665    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="545bba39-549f-4394-a079-60dea15f4947" containerName="kserve-container"
Sep 16 19:33:40.546868 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.546673    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5793b0e-0970-4df1-b61f-6c0812edad60" containerName="storage-initializer"
Sep 16 19:33:40.546868 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.546685    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="545bba39-549f-4394-a079-60dea15f4947" containerName="kube-rbac-proxy"
Sep 16 19:33:40.546868 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.546756    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5793b0e-0970-4df1-b61f-6c0812edad60" containerName="storage-initializer"
Sep 16 19:33:40.546868 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.546762    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5793b0e-0970-4df1-b61f-6c0812edad60" containerName="storage-initializer"
Sep 16 19:33:40.550547 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.550516    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"
Sep 16 19:33:40.553909 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.553343    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8gklb\""
Sep 16 19:33:40.553909 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.553612    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:33:40.553909 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.553640    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:33:40.554854 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.554444    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:33:40.554854 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.554513    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:33:40.554854 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.554452    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 16 19:33:40.554854 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.554812    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Sep 16 19:33:40.560925 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.560905    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"]
Sep 16 19:33:40.597587 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.597568    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hffwt\" (UniqueName: \"kubernetes.io/projected/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-kube-api-access-hffwt\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn\" (UID: \"49d45fcd-4405-4eb8-b007-a6f7e6fc9680\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"
Sep 16 19:33:40.597680 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.597599    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn\" (UID: \"49d45fcd-4405-4eb8-b007-a6f7e6fc9680\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"
Sep 16 19:33:40.597680 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.597639    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn\" (UID: \"49d45fcd-4405-4eb8-b007-a6f7e6fc9680\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"
Sep 16 19:33:40.597680 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.597662    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn\" (UID: \"49d45fcd-4405-4eb8-b007-a6f7e6fc9680\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"
Sep 16 19:33:40.597814 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.597699    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn\" (UID: \"49d45fcd-4405-4eb8-b007-a6f7e6fc9680\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"
Sep 16 19:33:40.698257 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.698234    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hffwt\" (UniqueName: \"kubernetes.io/projected/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-kube-api-access-hffwt\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn\" (UID: \"49d45fcd-4405-4eb8-b007-a6f7e6fc9680\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"
Sep 16 19:33:40.698367 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.698267    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn\" (UID: \"49d45fcd-4405-4eb8-b007-a6f7e6fc9680\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"
Sep 16 19:33:40.698367 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.698295    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn\" (UID: \"49d45fcd-4405-4eb8-b007-a6f7e6fc9680\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"
Sep 16 19:33:40.698367 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.698340    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn\" (UID: \"49d45fcd-4405-4eb8-b007-a6f7e6fc9680\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"
Sep 16 19:33:40.698593 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.698406    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn\" (UID: \"49d45fcd-4405-4eb8-b007-a6f7e6fc9680\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"
Sep 16 19:33:40.698593 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:33:40.698551    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-serving-cert: secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 16 19:33:40.698697 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.698598    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn\" (UID: \"49d45fcd-4405-4eb8-b007-a6f7e6fc9680\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"
Sep 16 19:33:40.698697 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:33:40.698620    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-proxy-tls podName:49d45fcd-4405-4eb8-b007-a6f7e6fc9680 nodeName:}" failed. No retries permitted until 2026-09-16 19:33:41.198598641 +0000 UTC m=+2931.274368266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-proxy-tls") pod "isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn" (UID: "49d45fcd-4405-4eb8-b007-a6f7e6fc9680") : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Sep 16 19:33:40.698967 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.698950    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn\" (UID: \"49d45fcd-4405-4eb8-b007-a6f7e6fc9680\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"
Sep 16 19:33:40.699020 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.699005    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn\" (UID: \"49d45fcd-4405-4eb8-b007-a6f7e6fc9680\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"
Sep 16 19:33:40.707739 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:40.707704    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hffwt\" (UniqueName: \"kubernetes.io/projected/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-kube-api-access-hffwt\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn\" (UID: \"49d45fcd-4405-4eb8-b007-a6f7e6fc9680\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"
Sep 16 19:33:41.200387 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:41.200345    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn\" (UID: \"49d45fcd-4405-4eb8-b007-a6f7e6fc9680\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"
Sep 16 19:33:41.202604 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:41.202583    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn\" (UID: \"49d45fcd-4405-4eb8-b007-a6f7e6fc9680\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"
Sep 16 19:33:41.463751 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:41.463670    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"
Sep 16 19:33:41.735812 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:41.735785    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"]
Sep 16 19:33:41.737502 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:33:41.737478    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d45fcd_4405_4eb8_b007_a6f7e6fc9680.slice/crio-36fb549f32832a38d3b61d8d3e821073f498bbb6182af2bcf5d6a233d7eb01ab WatchSource:0}: Error finding container 36fb549f32832a38d3b61d8d3e821073f498bbb6182af2bcf5d6a233d7eb01ab: Status 404 returned error can't find the container with id 36fb549f32832a38d3b61d8d3e821073f498bbb6182af2bcf5d6a233d7eb01ab
Sep 16 19:33:42.691086 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:42.691054    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn" event={"ID":"49d45fcd-4405-4eb8-b007-a6f7e6fc9680","Type":"ContainerStarted","Data":"71d44fabdb07133dd478db6686f8f5c6e8214f6b07b63be323702d59fc7b07c9"}
Sep 16 19:33:42.691271 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:42.691119    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn" event={"ID":"49d45fcd-4405-4eb8-b007-a6f7e6fc9680","Type":"ContainerStarted","Data":"36fb549f32832a38d3b61d8d3e821073f498bbb6182af2bcf5d6a233d7eb01ab"}
Sep 16 19:33:46.711026 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:46.710936    2554 generic.go:356] "Generic (PLEG): container finished" podID="49d45fcd-4405-4eb8-b007-a6f7e6fc9680" containerID="71d44fabdb07133dd478db6686f8f5c6e8214f6b07b63be323702d59fc7b07c9" exitCode=0
Sep 16 19:33:46.711026 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:46.710991    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn" event={"ID":"49d45fcd-4405-4eb8-b007-a6f7e6fc9680","Type":"ContainerDied","Data":"71d44fabdb07133dd478db6686f8f5c6e8214f6b07b63be323702d59fc7b07c9"}
Sep 16 19:33:47.717632 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:47.717594    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn" event={"ID":"49d45fcd-4405-4eb8-b007-a6f7e6fc9680","Type":"ContainerStarted","Data":"5ec170b5c38c39a7809a1827b67ecb6102e514da0866f9ee3b228dc3fccd70bf"}
Sep 16 19:33:47.718068 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:47.717679    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn" event={"ID":"49d45fcd-4405-4eb8-b007-a6f7e6fc9680","Type":"ContainerStarted","Data":"ad3649e97486a9b873993b8a0b6883371342f18fbc99fca46d47e0e31c4943c6"}
Sep 16 19:33:47.718068 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:47.717969    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"
Sep 16 19:33:47.718183 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:47.718115    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"
Sep 16 19:33:47.719419 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:47.719392    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn" podUID="49d45fcd-4405-4eb8-b007-a6f7e6fc9680" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 16 19:33:47.736785 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:47.736699    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn" podStartSLOduration=3.253417229 podStartE2EDuration="7.736684632s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.483267403s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:33:47.736404167 +0000 UTC m=+2937.812173797" watchObservedRunningTime="2026-09-16 19:33:47.736684632 +0000 UTC m=+2937.812454262"
Sep 16 19:33:48.721802 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:48.721761    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn" podUID="49d45fcd-4405-4eb8-b007-a6f7e6fc9680" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 16 19:33:49.331136 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:49.331093    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc"]
Sep 16 19:33:49.335897 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:49.335880    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc"
Sep 16 19:33:49.337810 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:49.337791    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Sep 16 19:33:49.338432 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:49.338417    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Sep 16 19:33:49.338853 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:49.338837    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 16 19:33:49.347333 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:49.347312    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc"]
Sep 16 19:33:49.366519 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:49.366493    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c5de929-95da-45d4-8d7c-43209a702f6f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc\" (UID: \"1c5de929-95da-45d4-8d7c-43209a702f6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc"
Sep 16 19:33:49.366602 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:49.366543    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c5de929-95da-45d4-8d7c-43209a702f6f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc\" (UID: \"1c5de929-95da-45d4-8d7c-43209a702f6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc"
Sep 16 19:33:49.366602 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:49.366565    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c5de929-95da-45d4-8d7c-43209a702f6f-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc\" (UID: \"1c5de929-95da-45d4-8d7c-43209a702f6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc"
Sep 16 19:33:49.366681 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:49.366608    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpg74\" (UniqueName: \"kubernetes.io/projected/1c5de929-95da-45d4-8d7c-43209a702f6f-kube-api-access-xpg74\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc\" (UID: \"1c5de929-95da-45d4-8d7c-43209a702f6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc"
Sep 16 19:33:49.467255 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:49.467225    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpg74\" (UniqueName: \"kubernetes.io/projected/1c5de929-95da-45d4-8d7c-43209a702f6f-kube-api-access-xpg74\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc\" (UID: \"1c5de929-95da-45d4-8d7c-43209a702f6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc"
Sep 16 19:33:49.467368 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:49.467264    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c5de929-95da-45d4-8d7c-43209a702f6f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc\" (UID: \"1c5de929-95da-45d4-8d7c-43209a702f6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc"
Sep 16 19:33:49.467368 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:49.467325    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c5de929-95da-45d4-8d7c-43209a702f6f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc\" (UID: \"1c5de929-95da-45d4-8d7c-43209a702f6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc"
Sep 16 19:33:49.467368 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:49.467353    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c5de929-95da-45d4-8d7c-43209a702f6f-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc\" (UID: \"1c5de929-95da-45d4-8d7c-43209a702f6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc"
Sep 16 19:33:49.467481 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:33:49.467458    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Sep 16 19:33:49.467531 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:33:49.467517    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5de929-95da-45d4-8d7c-43209a702f6f-proxy-tls podName:1c5de929-95da-45d4-8d7c-43209a702f6f nodeName:}" failed. No retries permitted until 2026-09-16 19:33:49.967496717 +0000 UTC m=+2940.043266340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1c5de929-95da-45d4-8d7c-43209a702f6f-proxy-tls") pod "isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc" (UID: "1c5de929-95da-45d4-8d7c-43209a702f6f") : secret "isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert" not found
Sep 16 19:33:49.467737 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:49.467707    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c5de929-95da-45d4-8d7c-43209a702f6f-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc\" (UID: \"1c5de929-95da-45d4-8d7c-43209a702f6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc"
Sep 16 19:33:49.468119 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:49.468097    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c5de929-95da-45d4-8d7c-43209a702f6f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc\" (UID: \"1c5de929-95da-45d4-8d7c-43209a702f6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc"
Sep 16 19:33:49.478833 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:49.478810    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpg74\" (UniqueName: \"kubernetes.io/projected/1c5de929-95da-45d4-8d7c-43209a702f6f-kube-api-access-xpg74\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc\" (UID: \"1c5de929-95da-45d4-8d7c-43209a702f6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc"
Sep 16 19:33:49.971635 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:49.971600    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c5de929-95da-45d4-8d7c-43209a702f6f-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc\" (UID: \"1c5de929-95da-45d4-8d7c-43209a702f6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc"
Sep 16 19:33:49.974033 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:49.974009    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c5de929-95da-45d4-8d7c-43209a702f6f-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc\" (UID: \"1c5de929-95da-45d4-8d7c-43209a702f6f\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc"
Sep 16 19:33:50.245748 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:50.245648    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc"
Sep 16 19:33:50.529220 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:50.529198    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc"]
Sep 16 19:33:50.530325 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:33:50.530298    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5de929_95da_45d4_8d7c_43209a702f6f.slice/crio-09dca3f295b171b0a5753eb73379f86ec50a61ba8daee4f5d02cbcf97dac0729 WatchSource:0}: Error finding container 09dca3f295b171b0a5753eb73379f86ec50a61ba8daee4f5d02cbcf97dac0729: Status 404 returned error can't find the container with id 09dca3f295b171b0a5753eb73379f86ec50a61ba8daee4f5d02cbcf97dac0729
Sep 16 19:33:50.535302 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:50.535282    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 16 19:33:50.733394 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:50.733359    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc" event={"ID":"1c5de929-95da-45d4-8d7c-43209a702f6f","Type":"ContainerStarted","Data":"d617ce99b8ea4dbda9f8bb3352e95e453f3bf38d87e2e1e1a46780b6c23388f5"}
Sep 16 19:33:50.733585 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:50.733417    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc" event={"ID":"1c5de929-95da-45d4-8d7c-43209a702f6f","Type":"ContainerStarted","Data":"09dca3f295b171b0a5753eb73379f86ec50a61ba8daee4f5d02cbcf97dac0729"}
Sep 16 19:33:53.726924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:53.726897    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"
Sep 16 19:33:53.727529 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:53.727505    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn" podUID="49d45fcd-4405-4eb8-b007-a6f7e6fc9680" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 16 19:33:56.764493 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:56.764467    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc_1c5de929-95da-45d4-8d7c-43209a702f6f/storage-initializer/0.log"
Sep 16 19:33:56.764929 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:56.764505    2554 generic.go:356] "Generic (PLEG): container finished" podID="1c5de929-95da-45d4-8d7c-43209a702f6f" containerID="d617ce99b8ea4dbda9f8bb3352e95e453f3bf38d87e2e1e1a46780b6c23388f5" exitCode=1
Sep 16 19:33:56.764929 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:56.764538    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc" event={"ID":"1c5de929-95da-45d4-8d7c-43209a702f6f","Type":"ContainerDied","Data":"d617ce99b8ea4dbda9f8bb3352e95e453f3bf38d87e2e1e1a46780b6c23388f5"}
Sep 16 19:33:57.771703 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:57.771677    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc_1c5de929-95da-45d4-8d7c-43209a702f6f/storage-initializer/0.log"
Sep 16 19:33:57.772080 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:57.771755    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc" event={"ID":"1c5de929-95da-45d4-8d7c-43209a702f6f","Type":"ContainerStarted","Data":"debffc473bcec66ca7fd99d4b77fe206ba89e719984e345856ced3ab29d9615f"}
Sep 16 19:33:59.283657 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:59.283624    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc"]
Sep 16 19:33:59.284134 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:33:59.283867    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc" podUID="1c5de929-95da-45d4-8d7c-43209a702f6f" containerName="storage-initializer" containerID="cri-o://debffc473bcec66ca7fd99d4b77fe206ba89e719984e345856ced3ab29d9615f" gracePeriod=30
Sep 16 19:34:03.727925 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:03.727888    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn" podUID="49d45fcd-4405-4eb8-b007-a6f7e6fc9680" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 16 19:34:03.801548 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:03.801526    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc_1c5de929-95da-45d4-8d7c-43209a702f6f/storage-initializer/1.log"
Sep 16 19:34:03.802024 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:03.802000    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc_1c5de929-95da-45d4-8d7c-43209a702f6f/storage-initializer/0.log"
Sep 16 19:34:03.802084 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:03.802046    2554 generic.go:356] "Generic (PLEG): container finished" podID="1c5de929-95da-45d4-8d7c-43209a702f6f" containerID="debffc473bcec66ca7fd99d4b77fe206ba89e719984e345856ced3ab29d9615f" exitCode=1
Sep 16 19:34:03.802132 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:03.802092    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc" event={"ID":"1c5de929-95da-45d4-8d7c-43209a702f6f","Type":"ContainerDied","Data":"debffc473bcec66ca7fd99d4b77fe206ba89e719984e345856ced3ab29d9615f"}
Sep 16 19:34:03.802167 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:03.802161    2554 scope.go:117] "RemoveContainer" containerID="d617ce99b8ea4dbda9f8bb3352e95e453f3bf38d87e2e1e1a46780b6c23388f5"
Sep 16 19:34:03.829304 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:03.829287    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc_1c5de929-95da-45d4-8d7c-43209a702f6f/storage-initializer/1.log"
Sep 16 19:34:03.832307 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:03.832293    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc"
Sep 16 19:34:03.888983 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:03.888931    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c5de929-95da-45d4-8d7c-43209a702f6f-kserve-provision-location\") pod \"1c5de929-95da-45d4-8d7c-43209a702f6f\" (UID: \"1c5de929-95da-45d4-8d7c-43209a702f6f\") "
Sep 16 19:34:03.888983 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:03.888982    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c5de929-95da-45d4-8d7c-43209a702f6f-proxy-tls\") pod \"1c5de929-95da-45d4-8d7c-43209a702f6f\" (UID: \"1c5de929-95da-45d4-8d7c-43209a702f6f\") "
Sep 16 19:34:03.889120 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:03.889021    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpg74\" (UniqueName: \"kubernetes.io/projected/1c5de929-95da-45d4-8d7c-43209a702f6f-kube-api-access-xpg74\") pod \"1c5de929-95da-45d4-8d7c-43209a702f6f\" (UID: \"1c5de929-95da-45d4-8d7c-43209a702f6f\") "
Sep 16 19:34:03.889120 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:03.889073    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c5de929-95da-45d4-8d7c-43209a702f6f-kube-rbac-proxy-sar-config\") pod \"1c5de929-95da-45d4-8d7c-43209a702f6f\" (UID: \"1c5de929-95da-45d4-8d7c-43209a702f6f\") "
Sep 16 19:34:03.889215 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:03.889191    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c5de929-95da-45d4-8d7c-43209a702f6f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1c5de929-95da-45d4-8d7c-43209a702f6f" (UID: "1c5de929-95da-45d4-8d7c-43209a702f6f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:03.889449 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:03.889422    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c5de929-95da-45d4-8d7c-43209a702f6f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1c5de929-95da-45d4-8d7c-43209a702f6f" (UID: "1c5de929-95da-45d4-8d7c-43209a702f6f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:34:03.890876 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:03.890858    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c5de929-95da-45d4-8d7c-43209a702f6f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1c5de929-95da-45d4-8d7c-43209a702f6f" (UID: "1c5de929-95da-45d4-8d7c-43209a702f6f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:34:03.890974 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:03.890958    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c5de929-95da-45d4-8d7c-43209a702f6f-kube-api-access-xpg74" (OuterVolumeSpecName: "kube-api-access-xpg74") pod "1c5de929-95da-45d4-8d7c-43209a702f6f" (UID: "1c5de929-95da-45d4-8d7c-43209a702f6f"). InnerVolumeSpecName "kube-api-access-xpg74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:34:03.990059 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:03.990039    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c5de929-95da-45d4-8d7c-43209a702f6f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:34:03.990059 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:03.990060    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c5de929-95da-45d4-8d7c-43209a702f6f-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:34:03.990177 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:03.990070    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c5de929-95da-45d4-8d7c-43209a702f6f-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:34:03.990177 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:03.990078    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpg74\" (UniqueName: \"kubernetes.io/projected/1c5de929-95da-45d4-8d7c-43209a702f6f-kube-api-access-xpg74\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:34:04.808752 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:04.808705    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc_1c5de929-95da-45d4-8d7c-43209a702f6f/storage-initializer/1.log"
Sep 16 19:34:04.809142 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:04.808822    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc"
Sep 16 19:34:04.809142 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:04.808822    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc" event={"ID":"1c5de929-95da-45d4-8d7c-43209a702f6f","Type":"ContainerDied","Data":"09dca3f295b171b0a5753eb73379f86ec50a61ba8daee4f5d02cbcf97dac0729"}
Sep 16 19:34:04.809142 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:04.808897    2554 scope.go:117] "RemoveContainer" containerID="debffc473bcec66ca7fd99d4b77fe206ba89e719984e345856ced3ab29d9615f"
Sep 16 19:34:04.840957 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:04.840937    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc"]
Sep 16 19:34:04.846931 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:04.846911    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-d74d4c69f-p5dtc"]
Sep 16 19:34:06.442806 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:06.442772    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c5de929-95da-45d4-8d7c-43209a702f6f" path="/var/lib/kubelet/pods/1c5de929-95da-45d4-8d7c-43209a702f6f/volumes"
Sep 16 19:34:13.727619 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:13.727533    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn" podUID="49d45fcd-4405-4eb8-b007-a6f7e6fc9680" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 16 19:34:23.728166 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:23.728131    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn" podUID="49d45fcd-4405-4eb8-b007-a6f7e6fc9680" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 16 19:34:33.728749 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:33.728701    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"
Sep 16 19:34:41.872887 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:41.872857    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"]
Sep 16 19:34:41.873300 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:41.873076    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn" podUID="49d45fcd-4405-4eb8-b007-a6f7e6fc9680" containerName="kserve-container" containerID="cri-o://ad3649e97486a9b873993b8a0b6883371342f18fbc99fca46d47e0e31c4943c6" gracePeriod=30
Sep 16 19:34:41.873300 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:41.873126    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn" podUID="49d45fcd-4405-4eb8-b007-a6f7e6fc9680" containerName="kube-rbac-proxy" containerID="cri-o://5ec170b5c38c39a7809a1827b67ecb6102e514da0866f9ee3b228dc3fccd70bf" gracePeriod=30
Sep 16 19:34:41.996008 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:41.995983    2554 generic.go:356] "Generic (PLEG): container finished" podID="49d45fcd-4405-4eb8-b007-a6f7e6fc9680" containerID="5ec170b5c38c39a7809a1827b67ecb6102e514da0866f9ee3b228dc3fccd70bf" exitCode=2
Sep 16 19:34:41.996120 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:41.996018    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn" event={"ID":"49d45fcd-4405-4eb8-b007-a6f7e6fc9680","Type":"ContainerDied","Data":"5ec170b5c38c39a7809a1827b67ecb6102e514da0866f9ee3b228dc3fccd70bf"}
Sep 16 19:34:43.722994 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:43.722956    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn" podUID="49d45fcd-4405-4eb8-b007-a6f7e6fc9680" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.65:8643/healthz\": dial tcp 10.134.0.65:8643: connect: connection refused"
Sep 16 19:34:43.727546 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:43.727521    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn" podUID="49d45fcd-4405-4eb8-b007-a6f7e6fc9680" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.65:8080: connect: connection refused"
Sep 16 19:34:44.308122 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:44.308097    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"
Sep 16 19:34:44.332870 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:44.332841    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-kube-rbac-proxy-sar-config\") pod \"49d45fcd-4405-4eb8-b007-a6f7e6fc9680\" (UID: \"49d45fcd-4405-4eb8-b007-a6f7e6fc9680\") "
Sep 16 19:34:44.332992 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:44.332906    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-cabundle-cert\") pod \"49d45fcd-4405-4eb8-b007-a6f7e6fc9680\" (UID: \"49d45fcd-4405-4eb8-b007-a6f7e6fc9680\") "
Sep 16 19:34:44.332992 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:44.332937    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-proxy-tls\") pod \"49d45fcd-4405-4eb8-b007-a6f7e6fc9680\" (UID: \"49d45fcd-4405-4eb8-b007-a6f7e6fc9680\") "
Sep 16 19:34:44.332992 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:44.332976    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-kserve-provision-location\") pod \"49d45fcd-4405-4eb8-b007-a6f7e6fc9680\" (UID: \"49d45fcd-4405-4eb8-b007-a6f7e6fc9680\") "
Sep 16 19:34:44.333141 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:44.333008    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hffwt\" (UniqueName: \"kubernetes.io/projected/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-kube-api-access-hffwt\") pod \"49d45fcd-4405-4eb8-b007-a6f7e6fc9680\" (UID: \"49d45fcd-4405-4eb8-b007-a6f7e6fc9680\") "
Sep 16 19:34:44.333225 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:44.333203    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "49d45fcd-4405-4eb8-b007-a6f7e6fc9680" (UID: "49d45fcd-4405-4eb8-b007-a6f7e6fc9680"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:34:44.333309 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:44.333243    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "49d45fcd-4405-4eb8-b007-a6f7e6fc9680" (UID: "49d45fcd-4405-4eb8-b007-a6f7e6fc9680"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:34:44.333369 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:44.333355    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:34:44.333420 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:44.333373    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:34:44.334973 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:44.334948    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "49d45fcd-4405-4eb8-b007-a6f7e6fc9680" (UID: "49d45fcd-4405-4eb8-b007-a6f7e6fc9680"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:34:44.335076 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:44.335051    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-kube-api-access-hffwt" (OuterVolumeSpecName: "kube-api-access-hffwt") pod "49d45fcd-4405-4eb8-b007-a6f7e6fc9680" (UID: "49d45fcd-4405-4eb8-b007-a6f7e6fc9680"). InnerVolumeSpecName "kube-api-access-hffwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:34:44.342851 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:44.342824    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "49d45fcd-4405-4eb8-b007-a6f7e6fc9680" (UID: "49d45fcd-4405-4eb8-b007-a6f7e6fc9680"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:34:44.434293 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:44.434235    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:34:44.434293 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:44.434264    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:34:44.434293 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:44.434273    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hffwt\" (UniqueName: \"kubernetes.io/projected/49d45fcd-4405-4eb8-b007-a6f7e6fc9680-kube-api-access-hffwt\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:34:45.011685 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:45.011648    2554 generic.go:356] "Generic (PLEG): container finished" podID="49d45fcd-4405-4eb8-b007-a6f7e6fc9680" containerID="ad3649e97486a9b873993b8a0b6883371342f18fbc99fca46d47e0e31c4943c6" exitCode=0
Sep 16 19:34:45.012224 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:45.011712    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"
Sep 16 19:34:45.012224 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:45.011742    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn" event={"ID":"49d45fcd-4405-4eb8-b007-a6f7e6fc9680","Type":"ContainerDied","Data":"ad3649e97486a9b873993b8a0b6883371342f18fbc99fca46d47e0e31c4943c6"}
Sep 16 19:34:45.012224 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:45.011803    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn" event={"ID":"49d45fcd-4405-4eb8-b007-a6f7e6fc9680","Type":"ContainerDied","Data":"36fb549f32832a38d3b61d8d3e821073f498bbb6182af2bcf5d6a233d7eb01ab"}
Sep 16 19:34:45.012224 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:45.011830    2554 scope.go:117] "RemoveContainer" containerID="5ec170b5c38c39a7809a1827b67ecb6102e514da0866f9ee3b228dc3fccd70bf"
Sep 16 19:34:45.020131 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:45.020112    2554 scope.go:117] "RemoveContainer" containerID="ad3649e97486a9b873993b8a0b6883371342f18fbc99fca46d47e0e31c4943c6"
Sep 16 19:34:45.027373 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:45.027336    2554 scope.go:117] "RemoveContainer" containerID="71d44fabdb07133dd478db6686f8f5c6e8214f6b07b63be323702d59fc7b07c9"
Sep 16 19:34:45.029081 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:45.029058    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"]
Sep 16 19:34:45.034547 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:45.034525    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-w2hgn"]
Sep 16 19:34:45.035510 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:45.035490    2554 scope.go:117] "RemoveContainer" containerID="5ec170b5c38c39a7809a1827b67ecb6102e514da0866f9ee3b228dc3fccd70bf"
Sep 16 19:34:45.035801 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:34:45.035779    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ec170b5c38c39a7809a1827b67ecb6102e514da0866f9ee3b228dc3fccd70bf\": container with ID starting with 5ec170b5c38c39a7809a1827b67ecb6102e514da0866f9ee3b228dc3fccd70bf not found: ID does not exist" containerID="5ec170b5c38c39a7809a1827b67ecb6102e514da0866f9ee3b228dc3fccd70bf"
Sep 16 19:34:45.035869 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:45.035811    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ec170b5c38c39a7809a1827b67ecb6102e514da0866f9ee3b228dc3fccd70bf"} err="failed to get container status \"5ec170b5c38c39a7809a1827b67ecb6102e514da0866f9ee3b228dc3fccd70bf\": rpc error: code = NotFound desc = could not find container \"5ec170b5c38c39a7809a1827b67ecb6102e514da0866f9ee3b228dc3fccd70bf\": container with ID starting with 5ec170b5c38c39a7809a1827b67ecb6102e514da0866f9ee3b228dc3fccd70bf not found: ID does not exist"
Sep 16 19:34:45.035869 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:45.035827    2554 scope.go:117] "RemoveContainer" containerID="ad3649e97486a9b873993b8a0b6883371342f18fbc99fca46d47e0e31c4943c6"
Sep 16 19:34:45.036079 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:34:45.036061    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad3649e97486a9b873993b8a0b6883371342f18fbc99fca46d47e0e31c4943c6\": container with ID starting with ad3649e97486a9b873993b8a0b6883371342f18fbc99fca46d47e0e31c4943c6 not found: ID does not exist" containerID="ad3649e97486a9b873993b8a0b6883371342f18fbc99fca46d47e0e31c4943c6"
Sep 16 19:34:45.036130 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:45.036084    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad3649e97486a9b873993b8a0b6883371342f18fbc99fca46d47e0e31c4943c6"} err="failed to get container status \"ad3649e97486a9b873993b8a0b6883371342f18fbc99fca46d47e0e31c4943c6\": rpc error: code = NotFound desc = could not find container \"ad3649e97486a9b873993b8a0b6883371342f18fbc99fca46d47e0e31c4943c6\": container with ID starting with ad3649e97486a9b873993b8a0b6883371342f18fbc99fca46d47e0e31c4943c6 not found: ID does not exist"
Sep 16 19:34:45.036130 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:45.036100    2554 scope.go:117] "RemoveContainer" containerID="71d44fabdb07133dd478db6686f8f5c6e8214f6b07b63be323702d59fc7b07c9"
Sep 16 19:34:45.036322 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:34:45.036308    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71d44fabdb07133dd478db6686f8f5c6e8214f6b07b63be323702d59fc7b07c9\": container with ID starting with 71d44fabdb07133dd478db6686f8f5c6e8214f6b07b63be323702d59fc7b07c9 not found: ID does not exist" containerID="71d44fabdb07133dd478db6686f8f5c6e8214f6b07b63be323702d59fc7b07c9"
Sep 16 19:34:45.036368 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:45.036325    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71d44fabdb07133dd478db6686f8f5c6e8214f6b07b63be323702d59fc7b07c9"} err="failed to get container status \"71d44fabdb07133dd478db6686f8f5c6e8214f6b07b63be323702d59fc7b07c9\": rpc error: code = NotFound desc = could not find container \"71d44fabdb07133dd478db6686f8f5c6e8214f6b07b63be323702d59fc7b07c9\": container with ID starting with 71d44fabdb07133dd478db6686f8f5c6e8214f6b07b63be323702d59fc7b07c9 not found: ID does not exist"
Sep 16 19:34:46.442794 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:46.442757    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49d45fcd-4405-4eb8-b007-a6f7e6fc9680" path="/var/lib/kubelet/pods/49d45fcd-4405-4eb8-b007-a6f7e6fc9680/volumes"
Sep 16 19:34:51.466835 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:51.466804    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/2.log"
Sep 16 19:34:51.467284 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:51.466899    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/2.log"
Sep 16 19:34:51.472607 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:51.472589    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovn-acl-logging/0.log"
Sep 16 19:34:51.472801 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:34:51.472612    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovn-acl-logging/0.log"
Sep 16 19:35:31.578448 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.578412    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx"]
Sep 16 19:35:31.578924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.578747    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c5de929-95da-45d4-8d7c-43209a702f6f" containerName="storage-initializer"
Sep 16 19:35:31.578924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.578758    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c5de929-95da-45d4-8d7c-43209a702f6f" containerName="storage-initializer"
Sep 16 19:35:31.578924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.578767    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49d45fcd-4405-4eb8-b007-a6f7e6fc9680" containerName="kserve-container"
Sep 16 19:35:31.578924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.578772    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="49d45fcd-4405-4eb8-b007-a6f7e6fc9680" containerName="kserve-container"
Sep 16 19:35:31.578924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.578779    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c5de929-95da-45d4-8d7c-43209a702f6f" containerName="storage-initializer"
Sep 16 19:35:31.578924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.578785    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c5de929-95da-45d4-8d7c-43209a702f6f" containerName="storage-initializer"
Sep 16 19:35:31.578924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.578801    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49d45fcd-4405-4eb8-b007-a6f7e6fc9680" containerName="kube-rbac-proxy"
Sep 16 19:35:31.578924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.578806    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="49d45fcd-4405-4eb8-b007-a6f7e6fc9680" containerName="kube-rbac-proxy"
Sep 16 19:35:31.578924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.578816    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49d45fcd-4405-4eb8-b007-a6f7e6fc9680" containerName="storage-initializer"
Sep 16 19:35:31.578924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.578822    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="49d45fcd-4405-4eb8-b007-a6f7e6fc9680" containerName="storage-initializer"
Sep 16 19:35:31.578924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.578874    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="49d45fcd-4405-4eb8-b007-a6f7e6fc9680" containerName="kserve-container"
Sep 16 19:35:31.578924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.578880    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c5de929-95da-45d4-8d7c-43209a702f6f" containerName="storage-initializer"
Sep 16 19:35:31.578924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.578887    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="49d45fcd-4405-4eb8-b007-a6f7e6fc9680" containerName="kube-rbac-proxy"
Sep 16 19:35:31.578924 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.578894    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c5de929-95da-45d4-8d7c-43209a702f6f" containerName="storage-initializer"
Sep 16 19:35:31.581787 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.581769    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx"
Sep 16 19:35:31.584857 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.584832    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Sep 16 19:35:31.586036 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.586016    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Sep 16 19:35:31.586149 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.586083    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:35:31.586149 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.586089    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 16 19:35:31.586280 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.586151    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8gklb\""
Sep 16 19:35:31.586734 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.586693    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:35:31.600129 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.600103    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx"]
Sep 16 19:35:31.703133 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.703110    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df5737be-f1ab-4161-b147-ddde56c566e3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx\" (UID: \"df5737be-f1ab-4161-b147-ddde56c566e3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx"
Sep 16 19:35:31.703220 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.703146    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df5737be-f1ab-4161-b147-ddde56c566e3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx\" (UID: \"df5737be-f1ab-4161-b147-ddde56c566e3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx"
Sep 16 19:35:31.703220 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.703170    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xrcq\" (UniqueName: \"kubernetes.io/projected/df5737be-f1ab-4161-b147-ddde56c566e3-kube-api-access-6xrcq\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx\" (UID: \"df5737be-f1ab-4161-b147-ddde56c566e3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx"
Sep 16 19:35:31.703302 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.703237    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df5737be-f1ab-4161-b147-ddde56c566e3-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx\" (UID: \"df5737be-f1ab-4161-b147-ddde56c566e3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx"
Sep 16 19:35:31.803952 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.803918    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df5737be-f1ab-4161-b147-ddde56c566e3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx\" (UID: \"df5737be-f1ab-4161-b147-ddde56c566e3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx"
Sep 16 19:35:31.804112 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.803962    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df5737be-f1ab-4161-b147-ddde56c566e3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx\" (UID: \"df5737be-f1ab-4161-b147-ddde56c566e3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx"
Sep 16 19:35:31.804112 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.803991    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xrcq\" (UniqueName: \"kubernetes.io/projected/df5737be-f1ab-4161-b147-ddde56c566e3-kube-api-access-6xrcq\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx\" (UID: \"df5737be-f1ab-4161-b147-ddde56c566e3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx"
Sep 16 19:35:31.804112 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.804027    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df5737be-f1ab-4161-b147-ddde56c566e3-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx\" (UID: \"df5737be-f1ab-4161-b147-ddde56c566e3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx"
Sep 16 19:35:31.804337 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.804317    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df5737be-f1ab-4161-b147-ddde56c566e3-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx\" (UID: \"df5737be-f1ab-4161-b147-ddde56c566e3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx"
Sep 16 19:35:31.804663 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.804633    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df5737be-f1ab-4161-b147-ddde56c566e3-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx\" (UID: \"df5737be-f1ab-4161-b147-ddde56c566e3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx"
Sep 16 19:35:31.806365 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.806346    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df5737be-f1ab-4161-b147-ddde56c566e3-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx\" (UID: \"df5737be-f1ab-4161-b147-ddde56c566e3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx"
Sep 16 19:35:31.811510 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.811490    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xrcq\" (UniqueName: \"kubernetes.io/projected/df5737be-f1ab-4161-b147-ddde56c566e3-kube-api-access-6xrcq\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx\" (UID: \"df5737be-f1ab-4161-b147-ddde56c566e3\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx"
Sep 16 19:35:31.891700 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:31.891643    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx"
Sep 16 19:35:32.164496 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:32.164473    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx"]
Sep 16 19:35:32.167255 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:35:32.167230    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf5737be_f1ab_4161_b147_ddde56c566e3.slice/crio-146e7cfb193dcff316d0b964ad4fcf3a57f22678cac2d6d50a7df1952fb00daa WatchSource:0}: Error finding container 146e7cfb193dcff316d0b964ad4fcf3a57f22678cac2d6d50a7df1952fb00daa: Status 404 returned error can't find the container with id 146e7cfb193dcff316d0b964ad4fcf3a57f22678cac2d6d50a7df1952fb00daa
Sep 16 19:35:32.216591 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:32.216569    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx" event={"ID":"df5737be-f1ab-4161-b147-ddde56c566e3","Type":"ContainerStarted","Data":"146e7cfb193dcff316d0b964ad4fcf3a57f22678cac2d6d50a7df1952fb00daa"}
Sep 16 19:35:33.223039 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:33.223005    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx" event={"ID":"df5737be-f1ab-4161-b147-ddde56c566e3","Type":"ContainerStarted","Data":"2ba8d5d055cb02b3a382391ddd7737f3a5d1deb71b094dcd312c38c494a5c5b6"}
Sep 16 19:35:37.248596 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:37.248569    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx_df5737be-f1ab-4161-b147-ddde56c566e3/storage-initializer/0.log"
Sep 16 19:35:37.248989 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:37.248608    2554 generic.go:356] "Generic (PLEG): container finished" podID="df5737be-f1ab-4161-b147-ddde56c566e3" containerID="2ba8d5d055cb02b3a382391ddd7737f3a5d1deb71b094dcd312c38c494a5c5b6" exitCode=1
Sep 16 19:35:37.248989 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:37.248641    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx" event={"ID":"df5737be-f1ab-4161-b147-ddde56c566e3","Type":"ContainerDied","Data":"2ba8d5d055cb02b3a382391ddd7737f3a5d1deb71b094dcd312c38c494a5c5b6"}
Sep 16 19:35:38.254436 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:38.254408    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx_df5737be-f1ab-4161-b147-ddde56c566e3/storage-initializer/0.log"
Sep 16 19:35:38.254820 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:38.254456    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx" event={"ID":"df5737be-f1ab-4161-b147-ddde56c566e3","Type":"ContainerStarted","Data":"392a372df3508bb13085156f593bd2fdfb6b42303cc086cd0d210306fa51bfd2"}
Sep 16 19:35:41.616581 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:41.616550    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx"]
Sep 16 19:35:41.616998 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:41.616764    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx" podUID="df5737be-f1ab-4161-b147-ddde56c566e3" containerName="storage-initializer" containerID="cri-o://392a372df3508bb13085156f593bd2fdfb6b42303cc086cd0d210306fa51bfd2" gracePeriod=30
Sep 16 19:35:41.743699 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:41.743679    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx_df5737be-f1ab-4161-b147-ddde56c566e3/storage-initializer/1.log"
Sep 16 19:35:41.744096 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:41.744077    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx_df5737be-f1ab-4161-b147-ddde56c566e3/storage-initializer/0.log"
Sep 16 19:35:41.746364 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:41.746349    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx"
Sep 16 19:35:41.783518 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:41.783499    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df5737be-f1ab-4161-b147-ddde56c566e3-kserve-provision-location\") pod \"df5737be-f1ab-4161-b147-ddde56c566e3\" (UID: \"df5737be-f1ab-4161-b147-ddde56c566e3\") "
Sep 16 19:35:41.783624 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:41.783522    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df5737be-f1ab-4161-b147-ddde56c566e3-kube-rbac-proxy-sar-config\") pod \"df5737be-f1ab-4161-b147-ddde56c566e3\" (UID: \"df5737be-f1ab-4161-b147-ddde56c566e3\") "
Sep 16 19:35:41.783624 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:41.783551    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6xrcq\" (UniqueName: \"kubernetes.io/projected/df5737be-f1ab-4161-b147-ddde56c566e3-kube-api-access-6xrcq\") pod \"df5737be-f1ab-4161-b147-ddde56c566e3\" (UID: \"df5737be-f1ab-4161-b147-ddde56c566e3\") "
Sep 16 19:35:41.783786 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:41.783623    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df5737be-f1ab-4161-b147-ddde56c566e3-proxy-tls\") pod \"df5737be-f1ab-4161-b147-ddde56c566e3\" (UID: \"df5737be-f1ab-4161-b147-ddde56c566e3\") "
Sep 16 19:35:41.783862 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:41.783780    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df5737be-f1ab-4161-b147-ddde56c566e3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "df5737be-f1ab-4161-b147-ddde56c566e3" (UID: "df5737be-f1ab-4161-b147-ddde56c566e3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:35:41.783862 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:41.783820    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df5737be-f1ab-4161-b147-ddde56c566e3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "df5737be-f1ab-4161-b147-ddde56c566e3" (UID: "df5737be-f1ab-4161-b147-ddde56c566e3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:35:41.785558 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:41.785540    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df5737be-f1ab-4161-b147-ddde56c566e3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "df5737be-f1ab-4161-b147-ddde56c566e3" (UID: "df5737be-f1ab-4161-b147-ddde56c566e3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:35:41.785661 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:41.785589    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df5737be-f1ab-4161-b147-ddde56c566e3-kube-api-access-6xrcq" (OuterVolumeSpecName: "kube-api-access-6xrcq") pod "df5737be-f1ab-4161-b147-ddde56c566e3" (UID: "df5737be-f1ab-4161-b147-ddde56c566e3"). InnerVolumeSpecName "kube-api-access-6xrcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:35:41.884557 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:41.884534    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df5737be-f1ab-4161-b147-ddde56c566e3-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:35:41.884557 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:41.884557    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/df5737be-f1ab-4161-b147-ddde56c566e3-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:35:41.884698 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:41.884567    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df5737be-f1ab-4161-b147-ddde56c566e3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:35:41.884698 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:41.884576    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xrcq\" (UniqueName: \"kubernetes.io/projected/df5737be-f1ab-4161-b147-ddde56c566e3-kube-api-access-6xrcq\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:35:42.273798 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.273771    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx_df5737be-f1ab-4161-b147-ddde56c566e3/storage-initializer/1.log"
Sep 16 19:35:42.274175 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.274159    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx_df5737be-f1ab-4161-b147-ddde56c566e3/storage-initializer/0.log"
Sep 16 19:35:42.274250 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.274193    2554 generic.go:356] "Generic (PLEG): container finished" podID="df5737be-f1ab-4161-b147-ddde56c566e3" containerID="392a372df3508bb13085156f593bd2fdfb6b42303cc086cd0d210306fa51bfd2" exitCode=1
Sep 16 19:35:42.274250 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.274227    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx" event={"ID":"df5737be-f1ab-4161-b147-ddde56c566e3","Type":"ContainerDied","Data":"392a372df3508bb13085156f593bd2fdfb6b42303cc086cd0d210306fa51bfd2"}
Sep 16 19:35:42.274362 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.274266    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx"
Sep 16 19:35:42.274362 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.274287    2554 scope.go:117] "RemoveContainer" containerID="392a372df3508bb13085156f593bd2fdfb6b42303cc086cd0d210306fa51bfd2"
Sep 16 19:35:42.274462 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.274273    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx" event={"ID":"df5737be-f1ab-4161-b147-ddde56c566e3","Type":"ContainerDied","Data":"146e7cfb193dcff316d0b964ad4fcf3a57f22678cac2d6d50a7df1952fb00daa"}
Sep 16 19:35:42.282964 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.282950    2554 scope.go:117] "RemoveContainer" containerID="2ba8d5d055cb02b3a382391ddd7737f3a5d1deb71b094dcd312c38c494a5c5b6"
Sep 16 19:35:42.290131 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.290115    2554 scope.go:117] "RemoveContainer" containerID="392a372df3508bb13085156f593bd2fdfb6b42303cc086cd0d210306fa51bfd2"
Sep 16 19:35:42.290372 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:35:42.290355    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"392a372df3508bb13085156f593bd2fdfb6b42303cc086cd0d210306fa51bfd2\": container with ID starting with 392a372df3508bb13085156f593bd2fdfb6b42303cc086cd0d210306fa51bfd2 not found: ID does not exist" containerID="392a372df3508bb13085156f593bd2fdfb6b42303cc086cd0d210306fa51bfd2"
Sep 16 19:35:42.290415 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.290381    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"392a372df3508bb13085156f593bd2fdfb6b42303cc086cd0d210306fa51bfd2"} err="failed to get container status \"392a372df3508bb13085156f593bd2fdfb6b42303cc086cd0d210306fa51bfd2\": rpc error: code = NotFound desc = could not find container \"392a372df3508bb13085156f593bd2fdfb6b42303cc086cd0d210306fa51bfd2\": container with ID starting with 392a372df3508bb13085156f593bd2fdfb6b42303cc086cd0d210306fa51bfd2 not found: ID does not exist"
Sep 16 19:35:42.290415 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.290399    2554 scope.go:117] "RemoveContainer" containerID="2ba8d5d055cb02b3a382391ddd7737f3a5d1deb71b094dcd312c38c494a5c5b6"
Sep 16 19:35:42.290633 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:35:42.290618    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ba8d5d055cb02b3a382391ddd7737f3a5d1deb71b094dcd312c38c494a5c5b6\": container with ID starting with 2ba8d5d055cb02b3a382391ddd7737f3a5d1deb71b094dcd312c38c494a5c5b6 not found: ID does not exist" containerID="2ba8d5d055cb02b3a382391ddd7737f3a5d1deb71b094dcd312c38c494a5c5b6"
Sep 16 19:35:42.290748 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.290637    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ba8d5d055cb02b3a382391ddd7737f3a5d1deb71b094dcd312c38c494a5c5b6"} err="failed to get container status \"2ba8d5d055cb02b3a382391ddd7737f3a5d1deb71b094dcd312c38c494a5c5b6\": rpc error: code = NotFound desc = could not find container \"2ba8d5d055cb02b3a382391ddd7737f3a5d1deb71b094dcd312c38c494a5c5b6\": container with ID starting with 2ba8d5d055cb02b3a382391ddd7737f3a5d1deb71b094dcd312c38c494a5c5b6 not found: ID does not exist"
Sep 16 19:35:42.308479 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.308457    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx"]
Sep 16 19:35:42.314278 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.314256    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-5795f7f4-bp6xx"]
Sep 16 19:35:42.443402 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.443374    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df5737be-f1ab-4161-b147-ddde56c566e3" path="/var/lib/kubelet/pods/df5737be-f1ab-4161-b147-ddde56c566e3/volumes"
Sep 16 19:35:42.699971 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.699896    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"]
Sep 16 19:35:42.700320 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.700237    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df5737be-f1ab-4161-b147-ddde56c566e3" containerName="storage-initializer"
Sep 16 19:35:42.700320 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.700249    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="df5737be-f1ab-4161-b147-ddde56c566e3" containerName="storage-initializer"
Sep 16 19:35:42.700320 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.700313    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="df5737be-f1ab-4161-b147-ddde56c566e3" containerName="storage-initializer"
Sep 16 19:35:42.700423 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.700325    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="df5737be-f1ab-4161-b147-ddde56c566e3" containerName="storage-initializer"
Sep 16 19:35:42.700423 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.700376    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df5737be-f1ab-4161-b147-ddde56c566e3" containerName="storage-initializer"
Sep 16 19:35:42.700423 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.700382    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="df5737be-f1ab-4161-b147-ddde56c566e3" containerName="storage-initializer"
Sep 16 19:35:42.704934 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.704911    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"
Sep 16 19:35:42.707074 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.707055    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8gklb\""
Sep 16 19:35:42.707618 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.707597    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:35:42.707756 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.707622    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Sep 16 19:35:42.707756 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.707621    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:35:42.707756 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.707745    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:35:42.707999 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.707829    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 16 19:35:42.708065 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.708055    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 16 19:35:42.717008 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.716989    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"]
Sep 16 19:35:42.790496 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.790464    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa0e0358-a198-4576-83a0-503632c57494-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc\" (UID: \"fa0e0358-a198-4576-83a0-503632c57494\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"
Sep 16 19:35:42.790648 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.790507    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa0e0358-a198-4576-83a0-503632c57494-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc\" (UID: \"fa0e0358-a198-4576-83a0-503632c57494\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"
Sep 16 19:35:42.790648 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.790530    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9dzj\" (UniqueName: \"kubernetes.io/projected/fa0e0358-a198-4576-83a0-503632c57494-kube-api-access-r9dzj\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc\" (UID: \"fa0e0358-a198-4576-83a0-503632c57494\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"
Sep 16 19:35:42.790648 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.790604    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa0e0358-a198-4576-83a0-503632c57494-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc\" (UID: \"fa0e0358-a198-4576-83a0-503632c57494\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"
Sep 16 19:35:42.790824 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.790648    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa0e0358-a198-4576-83a0-503632c57494-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc\" (UID: \"fa0e0358-a198-4576-83a0-503632c57494\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"
Sep 16 19:35:42.891424 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.891394    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa0e0358-a198-4576-83a0-503632c57494-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc\" (UID: \"fa0e0358-a198-4576-83a0-503632c57494\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"
Sep 16 19:35:42.891614 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.891435    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa0e0358-a198-4576-83a0-503632c57494-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc\" (UID: \"fa0e0358-a198-4576-83a0-503632c57494\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"
Sep 16 19:35:42.891614 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.891472    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa0e0358-a198-4576-83a0-503632c57494-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc\" (UID: \"fa0e0358-a198-4576-83a0-503632c57494\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"
Sep 16 19:35:42.891614 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.891498    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa0e0358-a198-4576-83a0-503632c57494-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc\" (UID: \"fa0e0358-a198-4576-83a0-503632c57494\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"
Sep 16 19:35:42.891614 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.891518    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9dzj\" (UniqueName: \"kubernetes.io/projected/fa0e0358-a198-4576-83a0-503632c57494-kube-api-access-r9dzj\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc\" (UID: \"fa0e0358-a198-4576-83a0-503632c57494\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"
Sep 16 19:35:42.891846 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:35:42.891632    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-serving-cert: secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Sep 16 19:35:42.891846 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:35:42.891712    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa0e0358-a198-4576-83a0-503632c57494-proxy-tls podName:fa0e0358-a198-4576-83a0-503632c57494 nodeName:}" failed. No retries permitted until 2026-09-16 19:35:43.391687869 +0000 UTC m=+3053.467457490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fa0e0358-a198-4576-83a0-503632c57494-proxy-tls") pod "isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc" (UID: "fa0e0358-a198-4576-83a0-503632c57494") : secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Sep 16 19:35:42.891937 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.891917    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa0e0358-a198-4576-83a0-503632c57494-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc\" (UID: \"fa0e0358-a198-4576-83a0-503632c57494\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"
Sep 16 19:35:42.892235 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.892219    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa0e0358-a198-4576-83a0-503632c57494-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc\" (UID: \"fa0e0358-a198-4576-83a0-503632c57494\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"
Sep 16 19:35:42.892272 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.892219    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa0e0358-a198-4576-83a0-503632c57494-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc\" (UID: \"fa0e0358-a198-4576-83a0-503632c57494\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"
Sep 16 19:35:42.902377 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:42.902355    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9dzj\" (UniqueName: \"kubernetes.io/projected/fa0e0358-a198-4576-83a0-503632c57494-kube-api-access-r9dzj\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc\" (UID: \"fa0e0358-a198-4576-83a0-503632c57494\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"
Sep 16 19:35:43.396167 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:43.396141    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa0e0358-a198-4576-83a0-503632c57494-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc\" (UID: \"fa0e0358-a198-4576-83a0-503632c57494\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"
Sep 16 19:35:43.398410 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:43.398393    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa0e0358-a198-4576-83a0-503632c57494-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc\" (UID: \"fa0e0358-a198-4576-83a0-503632c57494\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"
Sep 16 19:35:43.615109 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:43.615070    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"
Sep 16 19:35:43.885130 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:43.885107    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"]
Sep 16 19:35:43.887237 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:35:43.887211    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0e0358_a198_4576_83a0_503632c57494.slice/crio-7c4e6f9848a3baad2db0b265ef446f5cca646f18c96d68ca5e88aac6d238cc3d WatchSource:0}: Error finding container 7c4e6f9848a3baad2db0b265ef446f5cca646f18c96d68ca5e88aac6d238cc3d: Status 404 returned error can't find the container with id 7c4e6f9848a3baad2db0b265ef446f5cca646f18c96d68ca5e88aac6d238cc3d
Sep 16 19:35:44.286553 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:44.286517    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc" event={"ID":"fa0e0358-a198-4576-83a0-503632c57494","Type":"ContainerStarted","Data":"3189c3f5cdbab8b59cf9498f85ae1eb069388daccff9c1140afe502da600863d"}
Sep 16 19:35:44.286710 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:44.286586    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc" event={"ID":"fa0e0358-a198-4576-83a0-503632c57494","Type":"ContainerStarted","Data":"7c4e6f9848a3baad2db0b265ef446f5cca646f18c96d68ca5e88aac6d238cc3d"}
Sep 16 19:35:48.305130 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:48.305099    2554 generic.go:356] "Generic (PLEG): container finished" podID="fa0e0358-a198-4576-83a0-503632c57494" containerID="3189c3f5cdbab8b59cf9498f85ae1eb069388daccff9c1140afe502da600863d" exitCode=0
Sep 16 19:35:48.305466 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:48.305167    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc" event={"ID":"fa0e0358-a198-4576-83a0-503632c57494","Type":"ContainerDied","Data":"3189c3f5cdbab8b59cf9498f85ae1eb069388daccff9c1140afe502da600863d"}
Sep 16 19:35:56.353571 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:56.353536    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc" event={"ID":"fa0e0358-a198-4576-83a0-503632c57494","Type":"ContainerStarted","Data":"cfab55c2aabe59bee190b61ad47c9b291002101390dd3a3c559e245d18151347"}
Sep 16 19:35:56.353973 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:56.353624    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc" event={"ID":"fa0e0358-a198-4576-83a0-503632c57494","Type":"ContainerStarted","Data":"b6cb60824b085787719e43692a4dfd0b99a42480c979cda0e6343fe2cbb0ef47"}
Sep 16 19:35:56.353973 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:56.353948    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"
Sep 16 19:35:56.354095 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:56.354075    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"
Sep 16 19:35:56.355376 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:56.355351    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc" podUID="fa0e0358-a198-4576-83a0-503632c57494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 16 19:35:57.359142 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:57.359089    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc" podUID="fa0e0358-a198-4576-83a0-503632c57494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 16 19:35:58.363113 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:35:58.363066    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc" podUID="fa0e0358-a198-4576-83a0-503632c57494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 16 19:36:03.368039 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:36:03.368010    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"
Sep 16 19:36:03.368512 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:36:03.368454    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc" podUID="fa0e0358-a198-4576-83a0-503632c57494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 16 19:36:03.389856 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:36:03.389807    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc" podStartSLOduration=10.574146477 podStartE2EDuration="21.389791788s" totalImagesPullingTime="7.007820937s" totalInitContainerRuntime="3.807824374s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:35:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:35:56.400985728 +0000 UTC m=+3066.476755360" watchObservedRunningTime="2026-09-16 19:36:03.389791788 +0000 UTC m=+3073.465561421"
Sep 16 19:36:13.368494 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:36:13.368456    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc" podUID="fa0e0358-a198-4576-83a0-503632c57494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 16 19:36:23.369221 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:36:23.369181    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc" podUID="fa0e0358-a198-4576-83a0-503632c57494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 16 19:36:33.368436 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:36:33.368389    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc" podUID="fa0e0358-a198-4576-83a0-503632c57494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 16 19:36:43.369345 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:36:43.369302    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc" podUID="fa0e0358-a198-4576-83a0-503632c57494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 16 19:36:53.368779 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:36:53.368707    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc" podUID="fa0e0358-a198-4576-83a0-503632c57494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 16 19:37:03.368555 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:03.368514    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc" podUID="fa0e0358-a198-4576-83a0-503632c57494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.68:8080: connect: connection refused"
Sep 16 19:37:13.369596 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:13.369519    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"
Sep 16 19:37:23.876572 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:23.876542    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"]
Sep 16 19:37:23.876959 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:23.876798    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc" podUID="fa0e0358-a198-4576-83a0-503632c57494" containerName="kserve-container" containerID="cri-o://b6cb60824b085787719e43692a4dfd0b99a42480c979cda0e6343fe2cbb0ef47" gracePeriod=30
Sep 16 19:37:23.876959 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:23.876824    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc" podUID="fa0e0358-a198-4576-83a0-503632c57494" containerName="kube-rbac-proxy" containerID="cri-o://cfab55c2aabe59bee190b61ad47c9b291002101390dd3a3c559e245d18151347" gracePeriod=30
Sep 16 19:37:24.752448 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:24.752417    2554 generic.go:356] "Generic (PLEG): container finished" podID="fa0e0358-a198-4576-83a0-503632c57494" containerID="cfab55c2aabe59bee190b61ad47c9b291002101390dd3a3c559e245d18151347" exitCode=2
Sep 16 19:37:24.752620 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:24.752498    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc" event={"ID":"fa0e0358-a198-4576-83a0-503632c57494","Type":"ContainerDied","Data":"cfab55c2aabe59bee190b61ad47c9b291002101390dd3a3c559e245d18151347"}
Sep 16 19:37:26.911016 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:26.910989    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"
Sep 16 19:37:26.955611 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:26.955587    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa0e0358-a198-4576-83a0-503632c57494-kube-rbac-proxy-sar-config\") pod \"fa0e0358-a198-4576-83a0-503632c57494\" (UID: \"fa0e0358-a198-4576-83a0-503632c57494\") "
Sep 16 19:37:26.955737 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:26.955627    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa0e0358-a198-4576-83a0-503632c57494-proxy-tls\") pod \"fa0e0358-a198-4576-83a0-503632c57494\" (UID: \"fa0e0358-a198-4576-83a0-503632c57494\") "
Sep 16 19:37:26.955737 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:26.955658    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9dzj\" (UniqueName: \"kubernetes.io/projected/fa0e0358-a198-4576-83a0-503632c57494-kube-api-access-r9dzj\") pod \"fa0e0358-a198-4576-83a0-503632c57494\" (UID: \"fa0e0358-a198-4576-83a0-503632c57494\") "
Sep 16 19:37:26.955737 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:26.955687    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa0e0358-a198-4576-83a0-503632c57494-cabundle-cert\") pod \"fa0e0358-a198-4576-83a0-503632c57494\" (UID: \"fa0e0358-a198-4576-83a0-503632c57494\") "
Sep 16 19:37:26.955851 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:26.955770    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa0e0358-a198-4576-83a0-503632c57494-kserve-provision-location\") pod \"fa0e0358-a198-4576-83a0-503632c57494\" (UID: \"fa0e0358-a198-4576-83a0-503632c57494\") "
Sep 16 19:37:26.955944 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:26.955916    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa0e0358-a198-4576-83a0-503632c57494-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fa0e0358-a198-4576-83a0-503632c57494" (UID: "fa0e0358-a198-4576-83a0-503632c57494"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:37:26.956126 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:26.956106    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa0e0358-a198-4576-83a0-503632c57494-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fa0e0358-a198-4576-83a0-503632c57494" (UID: "fa0e0358-a198-4576-83a0-503632c57494"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:37:26.956201 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:26.956152    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa0e0358-a198-4576-83a0-503632c57494-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fa0e0358-a198-4576-83a0-503632c57494" (UID: "fa0e0358-a198-4576-83a0-503632c57494"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:37:26.957625 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:26.957604    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa0e0358-a198-4576-83a0-503632c57494-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fa0e0358-a198-4576-83a0-503632c57494" (UID: "fa0e0358-a198-4576-83a0-503632c57494"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:37:26.957705 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:26.957646    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0e0358-a198-4576-83a0-503632c57494-kube-api-access-r9dzj" (OuterVolumeSpecName: "kube-api-access-r9dzj") pod "fa0e0358-a198-4576-83a0-503632c57494" (UID: "fa0e0358-a198-4576-83a0-503632c57494"). InnerVolumeSpecName "kube-api-access-r9dzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:37:27.056459 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:27.056376    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa0e0358-a198-4576-83a0-503632c57494-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:37:27.056459 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:27.056409    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa0e0358-a198-4576-83a0-503632c57494-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:37:27.056459 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:27.056421    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa0e0358-a198-4576-83a0-503632c57494-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:37:27.056459 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:27.056431    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9dzj\" (UniqueName: \"kubernetes.io/projected/fa0e0358-a198-4576-83a0-503632c57494-kube-api-access-r9dzj\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:37:27.056459 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:27.056439    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa0e0358-a198-4576-83a0-503632c57494-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:37:27.767483 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:27.767449    2554 generic.go:356] "Generic (PLEG): container finished" podID="fa0e0358-a198-4576-83a0-503632c57494" containerID="b6cb60824b085787719e43692a4dfd0b99a42480c979cda0e6343fe2cbb0ef47" exitCode=0
Sep 16 19:37:27.767684 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:27.767512    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"
Sep 16 19:37:27.767684 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:27.767523    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc" event={"ID":"fa0e0358-a198-4576-83a0-503632c57494","Type":"ContainerDied","Data":"b6cb60824b085787719e43692a4dfd0b99a42480c979cda0e6343fe2cbb0ef47"}
Sep 16 19:37:27.767684 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:27.767574    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc" event={"ID":"fa0e0358-a198-4576-83a0-503632c57494","Type":"ContainerDied","Data":"7c4e6f9848a3baad2db0b265ef446f5cca646f18c96d68ca5e88aac6d238cc3d"}
Sep 16 19:37:27.767684 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:27.767600    2554 scope.go:117] "RemoveContainer" containerID="cfab55c2aabe59bee190b61ad47c9b291002101390dd3a3c559e245d18151347"
Sep 16 19:37:27.776123 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:27.776108    2554 scope.go:117] "RemoveContainer" containerID="b6cb60824b085787719e43692a4dfd0b99a42480c979cda0e6343fe2cbb0ef47"
Sep 16 19:37:27.783431 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:27.783413    2554 scope.go:117] "RemoveContainer" containerID="3189c3f5cdbab8b59cf9498f85ae1eb069388daccff9c1140afe502da600863d"
Sep 16 19:37:27.788657 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:27.788633    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"]
Sep 16 19:37:27.791492 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:27.791206    2554 scope.go:117] "RemoveContainer" containerID="cfab55c2aabe59bee190b61ad47c9b291002101390dd3a3c559e245d18151347"
Sep 16 19:37:27.791647 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:37:27.791622    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfab55c2aabe59bee190b61ad47c9b291002101390dd3a3c559e245d18151347\": container with ID starting with cfab55c2aabe59bee190b61ad47c9b291002101390dd3a3c559e245d18151347 not found: ID does not exist" containerID="cfab55c2aabe59bee190b61ad47c9b291002101390dd3a3c559e245d18151347"
Sep 16 19:37:27.791828 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:27.791657    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfab55c2aabe59bee190b61ad47c9b291002101390dd3a3c559e245d18151347"} err="failed to get container status \"cfab55c2aabe59bee190b61ad47c9b291002101390dd3a3c559e245d18151347\": rpc error: code = NotFound desc = could not find container \"cfab55c2aabe59bee190b61ad47c9b291002101390dd3a3c559e245d18151347\": container with ID starting with cfab55c2aabe59bee190b61ad47c9b291002101390dd3a3c559e245d18151347 not found: ID does not exist"
Sep 16 19:37:27.791828 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:27.791681    2554 scope.go:117] "RemoveContainer" containerID="b6cb60824b085787719e43692a4dfd0b99a42480c979cda0e6343fe2cbb0ef47"
Sep 16 19:37:27.792017 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:37:27.791992    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6cb60824b085787719e43692a4dfd0b99a42480c979cda0e6343fe2cbb0ef47\": container with ID starting with b6cb60824b085787719e43692a4dfd0b99a42480c979cda0e6343fe2cbb0ef47 not found: ID does not exist" containerID="b6cb60824b085787719e43692a4dfd0b99a42480c979cda0e6343fe2cbb0ef47"
Sep 16 19:37:27.792085 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:27.792029    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6cb60824b085787719e43692a4dfd0b99a42480c979cda0e6343fe2cbb0ef47"} err="failed to get container status \"b6cb60824b085787719e43692a4dfd0b99a42480c979cda0e6343fe2cbb0ef47\": rpc error: code = NotFound desc = could not find container \"b6cb60824b085787719e43692a4dfd0b99a42480c979cda0e6343fe2cbb0ef47\": container with ID starting with b6cb60824b085787719e43692a4dfd0b99a42480c979cda0e6343fe2cbb0ef47 not found: ID does not exist"
Sep 16 19:37:27.792085 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:27.792053    2554 scope.go:117] "RemoveContainer" containerID="3189c3f5cdbab8b59cf9498f85ae1eb069388daccff9c1140afe502da600863d"
Sep 16 19:37:27.792348 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:37:27.792332    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3189c3f5cdbab8b59cf9498f85ae1eb069388daccff9c1140afe502da600863d\": container with ID starting with 3189c3f5cdbab8b59cf9498f85ae1eb069388daccff9c1140afe502da600863d not found: ID does not exist" containerID="3189c3f5cdbab8b59cf9498f85ae1eb069388daccff9c1140afe502da600863d"
Sep 16 19:37:27.792407 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:27.792353    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3189c3f5cdbab8b59cf9498f85ae1eb069388daccff9c1140afe502da600863d"} err="failed to get container status \"3189c3f5cdbab8b59cf9498f85ae1eb069388daccff9c1140afe502da600863d\": rpc error: code = NotFound desc = could not find container \"3189c3f5cdbab8b59cf9498f85ae1eb069388daccff9c1140afe502da600863d\": container with ID starting with 3189c3f5cdbab8b59cf9498f85ae1eb069388daccff9c1140afe502da600863d not found: ID does not exist"
Sep 16 19:37:27.793091 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:27.793075    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-5h9rc"]
Sep 16 19:37:28.448884 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:37:28.444768    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa0e0358-a198-4576-83a0-503632c57494" path="/var/lib/kubelet/pods/fa0e0358-a198-4576-83a0-503632c57494/volumes"
Sep 16 19:39:51.502479 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:51.502441    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/2.log"
Sep 16 19:39:51.503200 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:51.503178    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/2.log"
Sep 16 19:39:51.508420 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:51.508394    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovn-acl-logging/0.log"
Sep 16 19:39:51.509078 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:51.509060    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovn-acl-logging/0.log"
Sep 16 19:39:54.875935 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:54.875896    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf"]
Sep 16 19:39:54.876323 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:54.876234    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa0e0358-a198-4576-83a0-503632c57494" containerName="kube-rbac-proxy"
Sep 16 19:39:54.876323 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:54.876245    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa0e0358-a198-4576-83a0-503632c57494" containerName="kube-rbac-proxy"
Sep 16 19:39:54.876323 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:54.876262    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa0e0358-a198-4576-83a0-503632c57494" containerName="storage-initializer"
Sep 16 19:39:54.876323 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:54.876267    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa0e0358-a198-4576-83a0-503632c57494" containerName="storage-initializer"
Sep 16 19:39:54.876323 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:54.876280    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa0e0358-a198-4576-83a0-503632c57494" containerName="kserve-container"
Sep 16 19:39:54.876323 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:54.876286    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa0e0358-a198-4576-83a0-503632c57494" containerName="kserve-container"
Sep 16 19:39:54.876523 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:54.876340    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa0e0358-a198-4576-83a0-503632c57494" containerName="kserve-container"
Sep 16 19:39:54.876523 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:54.876348    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa0e0358-a198-4576-83a0-503632c57494" containerName="kube-rbac-proxy"
Sep 16 19:39:54.879563 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:54.879546    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf"
Sep 16 19:39:54.881533 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:54.881508    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-4c492e-predictor-serving-cert\""
Sep 16 19:39:54.881678 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:54.881629    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-4c492e\""
Sep 16 19:39:54.881678 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:54.881633    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 16 19:39:54.881967 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:54.881951    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 16 19:39:54.882030 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:54.881977    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 16 19:39:54.882096 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:54.882040    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-4c492e-kube-rbac-proxy-sar-config\""
Sep 16 19:39:54.882153 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:54.882112    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-4c492e-dockercfg-n7j9s\""
Sep 16 19:39:54.889028 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:54.889003    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf"]
Sep 16 19:39:54.899856 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:54.899831    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-proxy-tls\") pod \"isvc-secondary-4c492e-predictor-84844d88fb-xmhdf\" (UID: \"e7e5fe12-f3c0-4864-9cdb-cc35831def1d\") " pod="kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf"
Sep 16 19:39:54.900005 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:54.899874    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-kserve-provision-location\") pod \"isvc-secondary-4c492e-predictor-84844d88fb-xmhdf\" (UID: \"e7e5fe12-f3c0-4864-9cdb-cc35831def1d\") " pod="kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf"
Sep 16 19:39:54.900005 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:54.899904    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-cabundle-cert\") pod \"isvc-secondary-4c492e-predictor-84844d88fb-xmhdf\" (UID: \"e7e5fe12-f3c0-4864-9cdb-cc35831def1d\") " pod="kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf"
Sep 16 19:39:54.900005 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:54.899973    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-4c492e-predictor-84844d88fb-xmhdf\" (UID: \"e7e5fe12-f3c0-4864-9cdb-cc35831def1d\") " pod="kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf"
Sep 16 19:39:54.900161 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:54.900007    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp882\" (UniqueName: \"kubernetes.io/projected/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-kube-api-access-gp882\") pod \"isvc-secondary-4c492e-predictor-84844d88fb-xmhdf\" (UID: \"e7e5fe12-f3c0-4864-9cdb-cc35831def1d\") " pod="kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf"
Sep 16 19:39:55.001441 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:55.001400    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-proxy-tls\") pod \"isvc-secondary-4c492e-predictor-84844d88fb-xmhdf\" (UID: \"e7e5fe12-f3c0-4864-9cdb-cc35831def1d\") " pod="kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf"
Sep 16 19:39:55.001441 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:55.001446    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-kserve-provision-location\") pod \"isvc-secondary-4c492e-predictor-84844d88fb-xmhdf\" (UID: \"e7e5fe12-f3c0-4864-9cdb-cc35831def1d\") " pod="kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf"
Sep 16 19:39:55.001713 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:55.001472    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-cabundle-cert\") pod \"isvc-secondary-4c492e-predictor-84844d88fb-xmhdf\" (UID: \"e7e5fe12-f3c0-4864-9cdb-cc35831def1d\") " pod="kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf"
Sep 16 19:39:55.001713 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:55.001504    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-4c492e-predictor-84844d88fb-xmhdf\" (UID: \"e7e5fe12-f3c0-4864-9cdb-cc35831def1d\") " pod="kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf"
Sep 16 19:39:55.001713 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:55.001537    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp882\" (UniqueName: \"kubernetes.io/projected/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-kube-api-access-gp882\") pod \"isvc-secondary-4c492e-predictor-84844d88fb-xmhdf\" (UID: \"e7e5fe12-f3c0-4864-9cdb-cc35831def1d\") " pod="kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf"
Sep 16 19:39:55.001910 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:55.001896    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-kserve-provision-location\") pod \"isvc-secondary-4c492e-predictor-84844d88fb-xmhdf\" (UID: \"e7e5fe12-f3c0-4864-9cdb-cc35831def1d\") " pod="kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf"
Sep 16 19:39:55.002205 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:55.002178    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-4c492e-predictor-84844d88fb-xmhdf\" (UID: \"e7e5fe12-f3c0-4864-9cdb-cc35831def1d\") " pod="kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf"
Sep 16 19:39:55.002334 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:55.002282    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-cabundle-cert\") pod \"isvc-secondary-4c492e-predictor-84844d88fb-xmhdf\" (UID: \"e7e5fe12-f3c0-4864-9cdb-cc35831def1d\") " pod="kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf"
Sep 16 19:39:55.004008 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:55.003986    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-proxy-tls\") pod \"isvc-secondary-4c492e-predictor-84844d88fb-xmhdf\" (UID: \"e7e5fe12-f3c0-4864-9cdb-cc35831def1d\") " pod="kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf"
Sep 16 19:39:55.009263 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:55.009243    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp882\" (UniqueName: \"kubernetes.io/projected/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-kube-api-access-gp882\") pod \"isvc-secondary-4c492e-predictor-84844d88fb-xmhdf\" (UID: \"e7e5fe12-f3c0-4864-9cdb-cc35831def1d\") " pod="kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf"
Sep 16 19:39:55.189698 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:55.189607    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf"
Sep 16 19:39:55.466615 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:55.466439    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf"]
Sep 16 19:39:55.469331 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:39:55.469296    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e5fe12_f3c0_4864_9cdb_cc35831def1d.slice/crio-64f088a411a75dacc87319ab3edfa9fe31dfd055e1406ead7f7cd5da2344fc29 WatchSource:0}: Error finding container 64f088a411a75dacc87319ab3edfa9fe31dfd055e1406ead7f7cd5da2344fc29: Status 404 returned error can't find the container with id 64f088a411a75dacc87319ab3edfa9fe31dfd055e1406ead7f7cd5da2344fc29
Sep 16 19:39:56.410424 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:56.410391    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf" event={"ID":"e7e5fe12-f3c0-4864-9cdb-cc35831def1d","Type":"ContainerStarted","Data":"085376368b355479f1d6fbf23b3357752be8c70ae5e21be0bff5e1d984c6fcf7"}
Sep 16 19:39:56.410834 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:56.410448    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf" event={"ID":"e7e5fe12-f3c0-4864-9cdb-cc35831def1d","Type":"ContainerStarted","Data":"64f088a411a75dacc87319ab3edfa9fe31dfd055e1406ead7f7cd5da2344fc29"}
Sep 16 19:39:59.426399 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:59.426321    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-4c492e-predictor-84844d88fb-xmhdf_e7e5fe12-f3c0-4864-9cdb-cc35831def1d/storage-initializer/0.log"
Sep 16 19:39:59.426399 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:59.426359    2554 generic.go:356] "Generic (PLEG): container finished" podID="e7e5fe12-f3c0-4864-9cdb-cc35831def1d" containerID="085376368b355479f1d6fbf23b3357752be8c70ae5e21be0bff5e1d984c6fcf7" exitCode=1
Sep 16 19:39:59.426872 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:39:59.426440    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf" event={"ID":"e7e5fe12-f3c0-4864-9cdb-cc35831def1d","Type":"ContainerDied","Data":"085376368b355479f1d6fbf23b3357752be8c70ae5e21be0bff5e1d984c6fcf7"}
Sep 16 19:40:00.432649 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:00.432615    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-4c492e-predictor-84844d88fb-xmhdf_e7e5fe12-f3c0-4864-9cdb-cc35831def1d/storage-initializer/0.log"
Sep 16 19:40:00.433063 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:00.432668    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf" event={"ID":"e7e5fe12-f3c0-4864-9cdb-cc35831def1d","Type":"ContainerStarted","Data":"a2a204104bccd43fca1f75da209c01b8c5c5ee16b0087c6745908127d24a04e3"}
Sep 16 19:40:03.447520 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:03.447486    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-4c492e-predictor-84844d88fb-xmhdf_e7e5fe12-f3c0-4864-9cdb-cc35831def1d/storage-initializer/1.log"
Sep 16 19:40:03.447933 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:03.447916    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-4c492e-predictor-84844d88fb-xmhdf_e7e5fe12-f3c0-4864-9cdb-cc35831def1d/storage-initializer/0.log"
Sep 16 19:40:03.447980 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:03.447951    2554 generic.go:356] "Generic (PLEG): container finished" podID="e7e5fe12-f3c0-4864-9cdb-cc35831def1d" containerID="a2a204104bccd43fca1f75da209c01b8c5c5ee16b0087c6745908127d24a04e3" exitCode=1
Sep 16 19:40:03.448017 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:03.448000    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf" event={"ID":"e7e5fe12-f3c0-4864-9cdb-cc35831def1d","Type":"ContainerDied","Data":"a2a204104bccd43fca1f75da209c01b8c5c5ee16b0087c6745908127d24a04e3"}
Sep 16 19:40:03.448059 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:03.448049    2554 scope.go:117] "RemoveContainer" containerID="085376368b355479f1d6fbf23b3357752be8c70ae5e21be0bff5e1d984c6fcf7"
Sep 16 19:40:03.448396 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:03.448377    2554 scope.go:117] "RemoveContainer" containerID="085376368b355479f1d6fbf23b3357752be8c70ae5e21be0bff5e1d984c6fcf7"
Sep 16 19:40:03.458862 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:40:03.458840    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-4c492e-predictor-84844d88fb-xmhdf_kserve-ci-e2e-test(e7e5fe12-f3c0-4864-9cdb-cc35831def1d)\"" pod="kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf" podUID="e7e5fe12-f3c0-4864-9cdb-cc35831def1d"
Sep 16 19:40:03.459037 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:40:03.459011    2554 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-4c492e-predictor-84844d88fb-xmhdf_kserve-ci-e2e-test_e7e5fe12-f3c0-4864-9cdb-cc35831def1d_0 in pod sandbox 64f088a411a75dacc87319ab3edfa9fe31dfd055e1406ead7f7cd5da2344fc29 from index: no such id: '085376368b355479f1d6fbf23b3357752be8c70ae5e21be0bff5e1d984c6fcf7'" containerID="085376368b355479f1d6fbf23b3357752be8c70ae5e21be0bff5e1d984c6fcf7"
Sep 16 19:40:03.459091 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:03.459049    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"085376368b355479f1d6fbf23b3357752be8c70ae5e21be0bff5e1d984c6fcf7"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-4c492e-predictor-84844d88fb-xmhdf_kserve-ci-e2e-test_e7e5fe12-f3c0-4864-9cdb-cc35831def1d_0 in pod sandbox 64f088a411a75dacc87319ab3edfa9fe31dfd055e1406ead7f7cd5da2344fc29 from index: no such id: '085376368b355479f1d6fbf23b3357752be8c70ae5e21be0bff5e1d984c6fcf7'"
Sep 16 19:40:04.453731 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:04.453690    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-4c492e-predictor-84844d88fb-xmhdf_e7e5fe12-f3c0-4864-9cdb-cc35831def1d/storage-initializer/1.log"
Sep 16 19:40:10.869539 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:10.869505    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf"]
Sep 16 19:40:10.995625 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:10.995600    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-4c492e-predictor-84844d88fb-xmhdf_e7e5fe12-f3c0-4864-9cdb-cc35831def1d/storage-initializer/1.log"
Sep 16 19:40:10.997895 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:10.997874    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf"
Sep 16 19:40:11.144279 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:11.144197    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-proxy-tls\") pod \"e7e5fe12-f3c0-4864-9cdb-cc35831def1d\" (UID: \"e7e5fe12-f3c0-4864-9cdb-cc35831def1d\") "
Sep 16 19:40:11.144279 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:11.144235    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gp882\" (UniqueName: \"kubernetes.io/projected/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-kube-api-access-gp882\") pod \"e7e5fe12-f3c0-4864-9cdb-cc35831def1d\" (UID: \"e7e5fe12-f3c0-4864-9cdb-cc35831def1d\") "
Sep 16 19:40:11.144485 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:11.144290    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-kube-rbac-proxy-sar-config\") pod \"e7e5fe12-f3c0-4864-9cdb-cc35831def1d\" (UID: \"e7e5fe12-f3c0-4864-9cdb-cc35831def1d\") "
Sep 16 19:40:11.144485 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:11.144390    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-kserve-provision-location\") pod \"e7e5fe12-f3c0-4864-9cdb-cc35831def1d\" (UID: \"e7e5fe12-f3c0-4864-9cdb-cc35831def1d\") "
Sep 16 19:40:11.144485 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:11.144458    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-cabundle-cert\") pod \"e7e5fe12-f3c0-4864-9cdb-cc35831def1d\" (UID: \"e7e5fe12-f3c0-4864-9cdb-cc35831def1d\") "
Sep 16 19:40:11.144702 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:11.144649    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e7e5fe12-f3c0-4864-9cdb-cc35831def1d" (UID: "e7e5fe12-f3c0-4864-9cdb-cc35831def1d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:40:11.144835 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:11.144678    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e7e5fe12-f3c0-4864-9cdb-cc35831def1d" (UID: "e7e5fe12-f3c0-4864-9cdb-cc35831def1d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:40:11.144903 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:11.144830    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e7e5fe12-f3c0-4864-9cdb-cc35831def1d" (UID: "e7e5fe12-f3c0-4864-9cdb-cc35831def1d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 16 19:40:11.146430 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:11.146410    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e7e5fe12-f3c0-4864-9cdb-cc35831def1d" (UID: "e7e5fe12-f3c0-4864-9cdb-cc35831def1d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 16 19:40:11.146430 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:11.146422    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-kube-api-access-gp882" (OuterVolumeSpecName: "kube-api-access-gp882") pod "e7e5fe12-f3c0-4864-9cdb-cc35831def1d" (UID: "e7e5fe12-f3c0-4864-9cdb-cc35831def1d"). InnerVolumeSpecName "kube-api-access-gp882". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:40:11.245152 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:11.245126    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-kserve-provision-location\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:40:11.245152 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:11.245150    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-cabundle-cert\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:40:11.245314 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:11.245161    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-proxy-tls\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:40:11.245314 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:11.245170    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp882\" (UniqueName: \"kubernetes.io/projected/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-kube-api-access-gp882\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:40:11.245314 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:11.245178    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e7e5fe12-f3c0-4864-9cdb-cc35831def1d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:40:11.484538 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:11.484461    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-4c492e-predictor-84844d88fb-xmhdf_e7e5fe12-f3c0-4864-9cdb-cc35831def1d/storage-initializer/1.log"
Sep 16 19:40:11.484685 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:11.484553    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf" event={"ID":"e7e5fe12-f3c0-4864-9cdb-cc35831def1d","Type":"ContainerDied","Data":"64f088a411a75dacc87319ab3edfa9fe31dfd055e1406ead7f7cd5da2344fc29"}
Sep 16 19:40:11.484685 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:11.484623    2554 scope.go:117] "RemoveContainer" containerID="a2a204104bccd43fca1f75da209c01b8c5c5ee16b0087c6745908127d24a04e3"
Sep 16 19:40:11.484685 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:11.484565    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf"
Sep 16 19:40:11.519865 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:11.519836    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf"]
Sep 16 19:40:11.524116 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:11.524090    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-4c492e-predictor-84844d88fb-xmhdf"]
Sep 16 19:40:12.443593 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:12.443560    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7e5fe12-f3c0-4864-9cdb-cc35831def1d" path="/var/lib/kubelet/pods/e7e5fe12-f3c0-4864-9cdb-cc35831def1d/volumes"
Sep 16 19:40:17.237943 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:17.237858    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-plvj7/must-gather-fxslj"]
Sep 16 19:40:17.238406 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:17.238223    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7e5fe12-f3c0-4864-9cdb-cc35831def1d" containerName="storage-initializer"
Sep 16 19:40:17.238406 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:17.238234    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7e5fe12-f3c0-4864-9cdb-cc35831def1d" containerName="storage-initializer"
Sep 16 19:40:17.238406 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:17.238312    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7e5fe12-f3c0-4864-9cdb-cc35831def1d" containerName="storage-initializer"
Sep 16 19:40:17.238406 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:17.238369    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7e5fe12-f3c0-4864-9cdb-cc35831def1d" containerName="storage-initializer"
Sep 16 19:40:17.238406 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:17.238375    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7e5fe12-f3c0-4864-9cdb-cc35831def1d" containerName="storage-initializer"
Sep 16 19:40:17.238574 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:17.238435    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7e5fe12-f3c0-4864-9cdb-cc35831def1d" containerName="storage-initializer"
Sep 16 19:40:17.242705 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:17.242684    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-plvj7/must-gather-fxslj"
Sep 16 19:40:17.245176 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:17.245157    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-plvj7\"/\"kube-root-ca.crt\""
Sep 16 19:40:17.245256 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:17.245198    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-plvj7\"/\"openshift-service-ca.crt\""
Sep 16 19:40:17.290875 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:17.290844    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-plvj7/must-gather-fxslj"]
Sep 16 19:40:17.396330 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:17.396297    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpzsb\" (UniqueName: \"kubernetes.io/projected/96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7-kube-api-access-rpzsb\") pod \"must-gather-fxslj\" (UID: \"96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7\") " pod="openshift-must-gather-plvj7/must-gather-fxslj"
Sep 16 19:40:17.396513 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:17.396378    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7-must-gather-output\") pod \"must-gather-fxslj\" (UID: \"96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7\") " pod="openshift-must-gather-plvj7/must-gather-fxslj"
Sep 16 19:40:17.497844 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:17.497753    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7-must-gather-output\") pod \"must-gather-fxslj\" (UID: \"96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7\") " pod="openshift-must-gather-plvj7/must-gather-fxslj"
Sep 16 19:40:17.497844 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:17.497813    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpzsb\" (UniqueName: \"kubernetes.io/projected/96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7-kube-api-access-rpzsb\") pod \"must-gather-fxslj\" (UID: \"96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7\") " pod="openshift-must-gather-plvj7/must-gather-fxslj"
Sep 16 19:40:17.498200 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:17.498176    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7-must-gather-output\") pod \"must-gather-fxslj\" (UID: \"96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7\") " pod="openshift-must-gather-plvj7/must-gather-fxslj"
Sep 16 19:40:17.506407 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:17.506379    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpzsb\" (UniqueName: \"kubernetes.io/projected/96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7-kube-api-access-rpzsb\") pod \"must-gather-fxslj\" (UID: \"96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7\") " pod="openshift-must-gather-plvj7/must-gather-fxslj"
Sep 16 19:40:17.560763 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:17.560730    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-plvj7/must-gather-fxslj"
Sep 16 19:40:17.827870 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:17.827845    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-plvj7/must-gather-fxslj"]
Sep 16 19:40:17.830515 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:40:17.830491    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96f339cd_c4e2_44bb_b8fb_f8980d1cc6e7.slice/crio-838a14546e314eabefe6ddbd30bfa545839501ec8c82da526883496d36c9b5ba WatchSource:0}: Error finding container 838a14546e314eabefe6ddbd30bfa545839501ec8c82da526883496d36c9b5ba: Status 404 returned error can't find the container with id 838a14546e314eabefe6ddbd30bfa545839501ec8c82da526883496d36c9b5ba
Sep 16 19:40:17.832214 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:17.832199    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 16 19:40:18.536371 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:18.536331    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-plvj7/must-gather-fxslj" event={"ID":"96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7","Type":"ContainerStarted","Data":"838a14546e314eabefe6ddbd30bfa545839501ec8c82da526883496d36c9b5ba"}
Sep 16 19:40:24.576489 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:24.576448    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-plvj7/must-gather-fxslj" event={"ID":"96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7","Type":"ContainerStarted","Data":"ee5abd2adf1070a10736f155c4cd4b539f27c874986aeba1743f132d59e72592"}
Sep 16 19:40:24.576881 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:24.576497    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-plvj7/must-gather-fxslj" event={"ID":"96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7","Type":"ContainerStarted","Data":"865b99427b12183c922c8cda44604b0c49f9d5d8ab50d52277b77af8dcfbb714"}
Sep 16 19:40:24.593492 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:24.593441    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-plvj7/must-gather-fxslj" podStartSLOduration=1.543470806 podStartE2EDuration="7.593424053s" totalImagesPullingTime="6.049953247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:40:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:40:24.592153056 +0000 UTC m=+3334.667922698" watchObservedRunningTime="2026-09-16 19:40:24.593424053 +0000 UTC m=+3334.669193684"
Sep 16 19:40:52.729325 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:52.729291    2554 generic.go:356] "Generic (PLEG): container finished" podID="96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7" containerID="865b99427b12183c922c8cda44604b0c49f9d5d8ab50d52277b77af8dcfbb714" exitCode=0
Sep 16 19:40:52.729804 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:52.729362    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-plvj7/must-gather-fxslj" event={"ID":"96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7","Type":"ContainerDied","Data":"865b99427b12183c922c8cda44604b0c49f9d5d8ab50d52277b77af8dcfbb714"}
Sep 16 19:40:52.729804 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:52.729677    2554 scope.go:117] "RemoveContainer" containerID="865b99427b12183c922c8cda44604b0c49f9d5d8ab50d52277b77af8dcfbb714"
Sep 16 19:40:53.265546 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:53.265516    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-plvj7_must-gather-fxslj_96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7/gather/0.log"
Sep 16 19:40:53.946929 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:53.946898    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t4xfr/must-gather-8fzz4"]
Sep 16 19:40:53.950419 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:53.950403    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t4xfr/must-gather-8fzz4"
Sep 16 19:40:53.952752 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:53.952729    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t4xfr\"/\"kube-root-ca.crt\""
Sep 16 19:40:53.952878 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:53.952735    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t4xfr\"/\"default-dockercfg-6fc9k\""
Sep 16 19:40:53.952945 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:53.952928    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t4xfr\"/\"openshift-service-ca.crt\""
Sep 16 19:40:53.959246 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:53.959225    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t4xfr/must-gather-8fzz4"]
Sep 16 19:40:54.047027 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:54.046991    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kll4t\" (UniqueName: \"kubernetes.io/projected/317beecd-d01e-46d0-9f01-01cddd4e4db2-kube-api-access-kll4t\") pod \"must-gather-8fzz4\" (UID: \"317beecd-d01e-46d0-9f01-01cddd4e4db2\") " pod="openshift-must-gather-t4xfr/must-gather-8fzz4"
Sep 16 19:40:54.047267 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:54.047169    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/317beecd-d01e-46d0-9f01-01cddd4e4db2-must-gather-output\") pod \"must-gather-8fzz4\" (UID: \"317beecd-d01e-46d0-9f01-01cddd4e4db2\") " pod="openshift-must-gather-t4xfr/must-gather-8fzz4"
Sep 16 19:40:54.147608 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:54.147570    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/317beecd-d01e-46d0-9f01-01cddd4e4db2-must-gather-output\") pod \"must-gather-8fzz4\" (UID: \"317beecd-d01e-46d0-9f01-01cddd4e4db2\") " pod="openshift-must-gather-t4xfr/must-gather-8fzz4"
Sep 16 19:40:54.147608 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:54.147607    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kll4t\" (UniqueName: \"kubernetes.io/projected/317beecd-d01e-46d0-9f01-01cddd4e4db2-kube-api-access-kll4t\") pod \"must-gather-8fzz4\" (UID: \"317beecd-d01e-46d0-9f01-01cddd4e4db2\") " pod="openshift-must-gather-t4xfr/must-gather-8fzz4"
Sep 16 19:40:54.147956 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:54.147933    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/317beecd-d01e-46d0-9f01-01cddd4e4db2-must-gather-output\") pod \"must-gather-8fzz4\" (UID: \"317beecd-d01e-46d0-9f01-01cddd4e4db2\") " pod="openshift-must-gather-t4xfr/must-gather-8fzz4"
Sep 16 19:40:54.155697 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:54.155671    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kll4t\" (UniqueName: \"kubernetes.io/projected/317beecd-d01e-46d0-9f01-01cddd4e4db2-kube-api-access-kll4t\") pod \"must-gather-8fzz4\" (UID: \"317beecd-d01e-46d0-9f01-01cddd4e4db2\") " pod="openshift-must-gather-t4xfr/must-gather-8fzz4"
Sep 16 19:40:54.259988 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:54.259957    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t4xfr/must-gather-8fzz4"
Sep 16 19:40:54.530319 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:54.530292    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t4xfr/must-gather-8fzz4"]
Sep 16 19:40:54.532620 ip-10-0-137-241 kubenswrapper[2554]: W0916 19:40:54.532588    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod317beecd_d01e_46d0_9f01_01cddd4e4db2.slice/crio-0ee4716eb70db760318c2657c0dc65b08e4a286ada971663e98ec192af613d3c WatchSource:0}: Error finding container 0ee4716eb70db760318c2657c0dc65b08e4a286ada971663e98ec192af613d3c: Status 404 returned error can't find the container with id 0ee4716eb70db760318c2657c0dc65b08e4a286ada971663e98ec192af613d3c
Sep 16 19:40:54.742701 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:54.742659    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t4xfr/must-gather-8fzz4" event={"ID":"317beecd-d01e-46d0-9f01-01cddd4e4db2","Type":"ContainerStarted","Data":"0ee4716eb70db760318c2657c0dc65b08e4a286ada971663e98ec192af613d3c"}
Sep 16 19:40:55.751892 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:55.751852    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t4xfr/must-gather-8fzz4" event={"ID":"317beecd-d01e-46d0-9f01-01cddd4e4db2","Type":"ContainerStarted","Data":"d6bf28103bb8ac0ecd7c73f3d9e888845d32769cd4d2f410957732235ba13694"}
Sep 16 19:40:55.752314 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:55.751910    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t4xfr/must-gather-8fzz4" event={"ID":"317beecd-d01e-46d0-9f01-01cddd4e4db2","Type":"ContainerStarted","Data":"05ca67eca000bf3212683a08cf36aec976be551eb627c996941a974973b5db66"}
Sep 16 19:40:55.767456 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:55.767404    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-t4xfr/must-gather-8fzz4" podStartSLOduration=1.933879484 podStartE2EDuration="2.767388333s" totalImagesPullingTime="833.508849ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-16 19:40:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:40:55.765696748 +0000 UTC m=+3365.841466379" watchObservedRunningTime="2026-09-16 19:40:55.767388333 +0000 UTC m=+3365.843157967"
Sep 16 19:40:56.909872 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:56.909841    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4pmm6_9f2230c8-6009-4227-8dbc-12e10ebe55f2/global-pull-secret-syncer/0.log"
Sep 16 19:40:57.108269 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:57.108240    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-k8kfm_972f8c3b-0fc7-4073-be0e-d98f28705085/konnectivity-agent/0.log"
Sep 16 19:40:57.165058 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:57.164987    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-241.ec2.internal_515eac2af796c2741f582b95e3ba4c7a/haproxy/0.log"
Sep 16 19:40:58.796419 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:58.796365    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-plvj7/must-gather-fxslj"]
Sep 16 19:40:58.799839 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:58.796609    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-plvj7/must-gather-fxslj" podUID="96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7" containerName="copy" containerID="cri-o://ee5abd2adf1070a10736f155c4cd4b539f27c874986aeba1743f132d59e72592" gracePeriod=2
Sep 16 19:40:58.800136 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:40:58.800103    2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-fxslj\" is forbidden: User \"system:node:ip-10-0-137-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-plvj7\": no relationship found between node 'ip-10-0-137-241.ec2.internal' and this object" podUID="96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7" pod="openshift-must-gather-plvj7/must-gather-fxslj"
Sep 16 19:40:58.805901 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:58.802496    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-plvj7/must-gather-fxslj"]
Sep 16 19:40:59.191585 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:59.191540    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-plvj7_must-gather-fxslj_96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7/copy/0.log"
Sep 16 19:40:59.195888 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:59.193894    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-plvj7/must-gather-fxslj"
Sep 16 19:40:59.196039 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:40:59.195879    2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-fxslj\" is forbidden: User \"system:node:ip-10-0-137-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-plvj7\": no relationship found between node 'ip-10-0-137-241.ec2.internal' and this object" podUID="96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7" pod="openshift-must-gather-plvj7/must-gather-fxslj"
Sep 16 19:40:59.309546 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:59.309504    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rpzsb\" (UniqueName: \"kubernetes.io/projected/96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7-kube-api-access-rpzsb\") pod \"96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7\" (UID: \"96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7\") "
Sep 16 19:40:59.309769 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:59.309583    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7-must-gather-output\") pod \"96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7\" (UID: \"96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7\") "
Sep 16 19:40:59.311016 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:59.310984    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7" (UID: "96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 16 19:40:59.313687 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:59.313655    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7-kube-api-access-rpzsb" (OuterVolumeSpecName: "kube-api-access-rpzsb") pod "96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7" (UID: "96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7"). InnerVolumeSpecName "kube-api-access-rpzsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 16 19:40:59.410409 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:59.410313    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpzsb\" (UniqueName: \"kubernetes.io/projected/96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7-kube-api-access-rpzsb\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:40:59.410409 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:59.410352    2554 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7-must-gather-output\") on node \"ip-10-0-137-241.ec2.internal\" DevicePath \"\""
Sep 16 19:40:59.778847 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:59.778813    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-plvj7_must-gather-fxslj_96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7/copy/0.log"
Sep 16 19:40:59.779292 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:59.779265    2554 generic.go:356] "Generic (PLEG): container finished" podID="96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7" containerID="ee5abd2adf1070a10736f155c4cd4b539f27c874986aeba1743f132d59e72592" exitCode=143
Sep 16 19:40:59.779422 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:59.779380    2554 scope.go:117] "RemoveContainer" containerID="ee5abd2adf1070a10736f155c4cd4b539f27c874986aeba1743f132d59e72592"
Sep 16 19:40:59.779542 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:59.779526    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-plvj7/must-gather-fxslj"
Sep 16 19:40:59.782751 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:40:59.782657    2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-fxslj\" is forbidden: User \"system:node:ip-10-0-137-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-plvj7\": no relationship found between node 'ip-10-0-137-241.ec2.internal' and this object" podUID="96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7" pod="openshift-must-gather-plvj7/must-gather-fxslj"
Sep 16 19:40:59.794326 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:59.793999    2554 scope.go:117] "RemoveContainer" containerID="865b99427b12183c922c8cda44604b0c49f9d5d8ab50d52277b77af8dcfbb714"
Sep 16 19:40:59.823268 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:59.816550    2554 scope.go:117] "RemoveContainer" containerID="ee5abd2adf1070a10736f155c4cd4b539f27c874986aeba1743f132d59e72592"
Sep 16 19:40:59.823268 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:40:59.821079    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee5abd2adf1070a10736f155c4cd4b539f27c874986aeba1743f132d59e72592\": container with ID starting with ee5abd2adf1070a10736f155c4cd4b539f27c874986aeba1743f132d59e72592 not found: ID does not exist" containerID="ee5abd2adf1070a10736f155c4cd4b539f27c874986aeba1743f132d59e72592"
Sep 16 19:40:59.823268 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:59.821125    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee5abd2adf1070a10736f155c4cd4b539f27c874986aeba1743f132d59e72592"} err="failed to get container status \"ee5abd2adf1070a10736f155c4cd4b539f27c874986aeba1743f132d59e72592\": rpc error: code = NotFound desc = could not find container \"ee5abd2adf1070a10736f155c4cd4b539f27c874986aeba1743f132d59e72592\": container with ID starting with ee5abd2adf1070a10736f155c4cd4b539f27c874986aeba1743f132d59e72592 not found: ID does not exist"
Sep 16 19:40:59.823268 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:59.821153    2554 scope.go:117] "RemoveContainer" containerID="865b99427b12183c922c8cda44604b0c49f9d5d8ab50d52277b77af8dcfbb714"
Sep 16 19:40:59.823268 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:40:59.822875    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"865b99427b12183c922c8cda44604b0c49f9d5d8ab50d52277b77af8dcfbb714\": container with ID starting with 865b99427b12183c922c8cda44604b0c49f9d5d8ab50d52277b77af8dcfbb714 not found: ID does not exist" containerID="865b99427b12183c922c8cda44604b0c49f9d5d8ab50d52277b77af8dcfbb714"
Sep 16 19:40:59.823268 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:40:59.822913    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"865b99427b12183c922c8cda44604b0c49f9d5d8ab50d52277b77af8dcfbb714"} err="failed to get container status \"865b99427b12183c922c8cda44604b0c49f9d5d8ab50d52277b77af8dcfbb714\": rpc error: code = NotFound desc = could not find container \"865b99427b12183c922c8cda44604b0c49f9d5d8ab50d52277b77af8dcfbb714\": container with ID starting with 865b99427b12183c922c8cda44604b0c49f9d5d8ab50d52277b77af8dcfbb714 not found: ID does not exist"
Sep 16 19:40:59.830869 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:40:59.830832    2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-fxslj\" is forbidden: User \"system:node:ip-10-0-137-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-plvj7\": no relationship found between node 'ip-10-0-137-241.ec2.internal' and this object" podUID="96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7" pod="openshift-must-gather-plvj7/must-gather-fxslj"
Sep 16 19:41:00.445895 ip-10-0-137-241 kubenswrapper[2554]: E0916 19:41:00.445856    2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-fxslj\" is forbidden: User \"system:node:ip-10-0-137-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-plvj7\": no relationship found between node 'ip-10-0-137-241.ec2.internal' and this object" podUID="96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7" pod="openshift-must-gather-plvj7/must-gather-fxslj"
Sep 16 19:41:00.451680 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:00.451241    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7" path="/var/lib/kubelet/pods/96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7/volumes"
Sep 16 19:41:01.104656 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:01.104627    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_70271f51-1ee3-4393-a70c-ede0c7fa4229/alertmanager/0.log"
Sep 16 19:41:01.139933 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:01.139907    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_70271f51-1ee3-4393-a70c-ede0c7fa4229/config-reloader/0.log"
Sep 16 19:41:01.214675 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:01.214649    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_70271f51-1ee3-4393-a70c-ede0c7fa4229/kube-rbac-proxy-web/0.log"
Sep 16 19:41:01.271934 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:01.271908    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_70271f51-1ee3-4393-a70c-ede0c7fa4229/kube-rbac-proxy/0.log"
Sep 16 19:41:01.308826 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:01.308795    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_70271f51-1ee3-4393-a70c-ede0c7fa4229/kube-rbac-proxy-metric/0.log"
Sep 16 19:41:01.342707 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:01.342340    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_70271f51-1ee3-4393-a70c-ede0c7fa4229/prom-label-proxy/0.log"
Sep 16 19:41:01.407524 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:01.407411    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_70271f51-1ee3-4393-a70c-ede0c7fa4229/init-config-reloader/0.log"
Sep 16 19:41:01.456194 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:01.456163    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cd954b897-ln8nz_b62bbe6f-7dcd-4295-a2d8-78d1dd7036b5/cluster-monitoring-operator/0.log"
Sep 16 19:41:01.487064 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:01.487015    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-s5zl7_6239a312-62d8-4ea0-b52a-d96f7a4d233c/kube-state-metrics/0.log"
Sep 16 19:41:01.514222 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:01.514195    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-s5zl7_6239a312-62d8-4ea0-b52a-d96f7a4d233c/kube-rbac-proxy-main/0.log"
Sep 16 19:41:01.547948 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:01.547374    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-s5zl7_6239a312-62d8-4ea0-b52a-d96f7a4d233c/kube-rbac-proxy-self/0.log"
Sep 16 19:41:01.589265 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:01.589235    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-597df9c44c-sqk79_72b0bbb3-86be-47da-a955-334ab24d276c/metrics-server/0.log"
Sep 16 19:41:01.623419 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:01.623389    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-5b9ms_43f2ae30-2608-438b-8be9-e87438990203/monitoring-plugin/0.log"
Sep 16 19:41:01.864552 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:01.864515    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9z9fb_c745eb77-57e1-4567-b68f-01e301940170/node-exporter/0.log"
Sep 16 19:41:01.890795 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:01.890763    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9z9fb_c745eb77-57e1-4567-b68f-01e301940170/kube-rbac-proxy/0.log"
Sep 16 19:41:01.917175 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:01.917145    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9z9fb_c745eb77-57e1-4567-b68f-01e301940170/init-textfile/0.log"
Sep 16 19:41:02.054179 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:02.054131    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5a316a8e-4e14-4478-a5f1-58974dcc57d5/prometheus/0.log"
Sep 16 19:41:02.075550 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:02.075477    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5a316a8e-4e14-4478-a5f1-58974dcc57d5/config-reloader/0.log"
Sep 16 19:41:02.101408 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:02.101376    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5a316a8e-4e14-4478-a5f1-58974dcc57d5/thanos-sidecar/0.log"
Sep 16 19:41:02.127649 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:02.127564    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5a316a8e-4e14-4478-a5f1-58974dcc57d5/kube-rbac-proxy-web/0.log"
Sep 16 19:41:02.155661 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:02.155635    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5a316a8e-4e14-4478-a5f1-58974dcc57d5/kube-rbac-proxy/0.log"
Sep 16 19:41:02.184228 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:02.184199    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5a316a8e-4e14-4478-a5f1-58974dcc57d5/kube-rbac-proxy-thanos/0.log"
Sep 16 19:41:02.213942 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:02.213909    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5a316a8e-4e14-4478-a5f1-58974dcc57d5/init-config-reloader/0.log"
Sep 16 19:41:02.302513 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:02.302481    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5864cd99d7-zn2th_b480ee02-4a14-4111-84f6-21920933edba/prometheus-operator-admission-webhook/0.log"
Sep 16 19:41:02.347461 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:02.347406    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65b8b8b754-fw6nc_9ace2021-c720-428d-8449-a06755fbc9a6/telemeter-client/0.log"
Sep 16 19:41:02.373255 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:02.373222    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65b8b8b754-fw6nc_9ace2021-c720-428d-8449-a06755fbc9a6/reload/0.log"
Sep 16 19:41:02.402796 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:02.402696    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65b8b8b754-fw6nc_9ace2021-c720-428d-8449-a06755fbc9a6/kube-rbac-proxy/0.log"
Sep 16 19:41:02.436189 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:02.436160    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-df7545c7c-qwct5_be8ef430-695c-4699-8ed3-46cfc365643e/thanos-query/0.log"
Sep 16 19:41:02.462900 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:02.462873    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-df7545c7c-qwct5_be8ef430-695c-4699-8ed3-46cfc365643e/kube-rbac-proxy-web/0.log"
Sep 16 19:41:02.494678 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:02.494644    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-df7545c7c-qwct5_be8ef430-695c-4699-8ed3-46cfc365643e/kube-rbac-proxy/0.log"
Sep 16 19:41:02.533112 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:02.533066    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-df7545c7c-qwct5_be8ef430-695c-4699-8ed3-46cfc365643e/prom-label-proxy/0.log"
Sep 16 19:41:02.558376 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:02.558331    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-df7545c7c-qwct5_be8ef430-695c-4699-8ed3-46cfc365643e/kube-rbac-proxy-rules/0.log"
Sep 16 19:41:02.583457 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:02.583428    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-df7545c7c-qwct5_be8ef430-695c-4699-8ed3-46cfc365643e/kube-rbac-proxy-metrics/0.log"
Sep 16 19:41:04.516811 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:04.516775    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/2.log"
Sep 16 19:41:04.521627 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:04.521601    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-696884fb9c-llwg5_3589130e-56b7-4ce2-ad13-55804eabd1b3/console-operator/3.log"
Sep 16 19:41:05.872319 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:05.872277    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t4xfr/perf-node-gather-daemonset-72pc5"]
Sep 16 19:41:05.872951 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:05.872818    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7" containerName="copy"
Sep 16 19:41:05.872951 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:05.872836    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7" containerName="copy"
Sep 16 19:41:05.872951 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:05.872871    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7" containerName="gather"
Sep 16 19:41:05.872951 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:05.872880    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7" containerName="gather"
Sep 16 19:41:05.873173 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:05.872966    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7" containerName="gather"
Sep 16 19:41:05.873173 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:05.872978    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="96f339cd-c4e2-44bb-b8fb-f8980d1cc6e7" containerName="copy"
Sep 16 19:41:05.876820 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:05.876789    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-72pc5"
Sep 16 19:41:05.886450 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:05.885971    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t4xfr/perf-node-gather-daemonset-72pc5"]
Sep 16 19:41:06.078868 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:06.078831    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a5135ae1-8ca7-4728-8654-1566c9ae3c7a-proc\") pod \"perf-node-gather-daemonset-72pc5\" (UID: \"a5135ae1-8ca7-4728-8654-1566c9ae3c7a\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-72pc5"
Sep 16 19:41:06.079060 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:06.078904    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a5135ae1-8ca7-4728-8654-1566c9ae3c7a-lib-modules\") pod \"perf-node-gather-daemonset-72pc5\" (UID: \"a5135ae1-8ca7-4728-8654-1566c9ae3c7a\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-72pc5"
Sep 16 19:41:06.079060 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:06.078962    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5135ae1-8ca7-4728-8654-1566c9ae3c7a-sys\") pod \"perf-node-gather-daemonset-72pc5\" (UID: \"a5135ae1-8ca7-4728-8654-1566c9ae3c7a\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-72pc5"
Sep 16 19:41:06.079060 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:06.079000    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a5135ae1-8ca7-4728-8654-1566c9ae3c7a-podres\") pod \"perf-node-gather-daemonset-72pc5\" (UID: \"a5135ae1-8ca7-4728-8654-1566c9ae3c7a\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-72pc5"
Sep 16 19:41:06.079060 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:06.079023    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvzfn\" (UniqueName: \"kubernetes.io/projected/a5135ae1-8ca7-4728-8654-1566c9ae3c7a-kube-api-access-kvzfn\") pod \"perf-node-gather-daemonset-72pc5\" (UID: \"a5135ae1-8ca7-4728-8654-1566c9ae3c7a\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-72pc5"
Sep 16 19:41:06.179603 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:06.179510    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a5135ae1-8ca7-4728-8654-1566c9ae3c7a-proc\") pod \"perf-node-gather-daemonset-72pc5\" (UID: \"a5135ae1-8ca7-4728-8654-1566c9ae3c7a\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-72pc5"
Sep 16 19:41:06.179603 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:06.179561    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a5135ae1-8ca7-4728-8654-1566c9ae3c7a-lib-modules\") pod \"perf-node-gather-daemonset-72pc5\" (UID: \"a5135ae1-8ca7-4728-8654-1566c9ae3c7a\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-72pc5"
Sep 16 19:41:06.179869 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:06.179607    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5135ae1-8ca7-4728-8654-1566c9ae3c7a-sys\") pod \"perf-node-gather-daemonset-72pc5\" (UID: \"a5135ae1-8ca7-4728-8654-1566c9ae3c7a\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-72pc5"
Sep 16 19:41:06.179869 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:06.179640    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a5135ae1-8ca7-4728-8654-1566c9ae3c7a-podres\") pod \"perf-node-gather-daemonset-72pc5\" (UID: \"a5135ae1-8ca7-4728-8654-1566c9ae3c7a\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-72pc5"
Sep 16 19:41:06.179869 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:06.179647    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a5135ae1-8ca7-4728-8654-1566c9ae3c7a-proc\") pod \"perf-node-gather-daemonset-72pc5\" (UID: \"a5135ae1-8ca7-4728-8654-1566c9ae3c7a\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-72pc5"
Sep 16 19:41:06.179869 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:06.179704    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a5135ae1-8ca7-4728-8654-1566c9ae3c7a-sys\") pod \"perf-node-gather-daemonset-72pc5\" (UID: \"a5135ae1-8ca7-4728-8654-1566c9ae3c7a\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-72pc5"
Sep 16 19:41:06.179869 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:06.179757    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvzfn\" (UniqueName: \"kubernetes.io/projected/a5135ae1-8ca7-4728-8654-1566c9ae3c7a-kube-api-access-kvzfn\") pod \"perf-node-gather-daemonset-72pc5\" (UID: \"a5135ae1-8ca7-4728-8654-1566c9ae3c7a\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-72pc5"
Sep 16 19:41:06.179869 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:06.179804    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a5135ae1-8ca7-4728-8654-1566c9ae3c7a-podres\") pod \"perf-node-gather-daemonset-72pc5\" (UID: \"a5135ae1-8ca7-4728-8654-1566c9ae3c7a\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-72pc5"
Sep 16 19:41:06.179869 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:06.179803    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a5135ae1-8ca7-4728-8654-1566c9ae3c7a-lib-modules\") pod \"perf-node-gather-daemonset-72pc5\" (UID: \"a5135ae1-8ca7-4728-8654-1566c9ae3c7a\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-72pc5"
Sep 16 19:41:06.187803 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:06.187708    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvzfn\" (UniqueName: \"kubernetes.io/projected/a5135ae1-8ca7-4728-8654-1566c9ae3c7a-kube-api-access-kvzfn\") pod \"perf-node-gather-daemonset-72pc5\" (UID: \"a5135ae1-8ca7-4728-8654-1566c9ae3c7a\") " pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-72pc5"
Sep 16 19:41:06.191692 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:06.191491    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-72pc5"
Sep 16 19:41:06.491231 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:06.491087    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t4xfr/perf-node-gather-daemonset-72pc5"]
Sep 16 19:41:06.587482 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:06.587462    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tjwj4_d0bc4f19-10ce-472d-b31b-dd58bb9d7afa/dns/0.log"
Sep 16 19:41:06.610402 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:06.610365    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tjwj4_d0bc4f19-10ce-472d-b31b-dd58bb9d7afa/kube-rbac-proxy/0.log"
Sep 16 19:41:06.731289 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:06.731251    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tdn77_8f16d955-3a40-4270-8810-9f50eecea21d/dns-node-resolver/0.log"
Sep 16 19:41:06.826140 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:06.826107    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-72pc5" event={"ID":"a5135ae1-8ca7-4728-8654-1566c9ae3c7a","Type":"ContainerStarted","Data":"7078961126a8539f308af560533d2a02d885505a72b44a2b5e15feba622f7ece"}
Sep 16 19:41:06.826402 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:06.826165    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-72pc5" event={"ID":"a5135ae1-8ca7-4728-8654-1566c9ae3c7a","Type":"ContainerStarted","Data":"7cbe4551bee0031bf0200c6b236a1058d4bf755da1cf28e0631596659080bd86"}
Sep 16 19:41:06.826402 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:06.826194    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-72pc5"
Sep 16 19:41:06.841528 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:06.841464    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-72pc5" podStartSLOduration=1.841442861 podStartE2EDuration="1.841442861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-16 19:41:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-16 19:41:06.841081568 +0000 UTC m=+3376.916851238" watchObservedRunningTime="2026-09-16 19:41:06.841442861 +0000 UTC m=+3376.917212493"
Sep 16 19:41:07.263705 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:07.263670    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-grtxh_e82386e7-000e-4693-aa95-d54f59be5f8b/node-ca/0.log"
Sep 16 19:41:08.451526 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:08.451498    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pbdzg_d7440848-ec21-400a-9143-2b6794067e8a/serve-healthcheck-canary/0.log"
Sep 16 19:41:08.932468 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:08.932439    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nrxcw_3900c7ee-b6e7-4edf-84a3-e44cb8c75b88/kube-rbac-proxy/0.log"
Sep 16 19:41:08.955866 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:08.955835    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nrxcw_3900c7ee-b6e7-4edf-84a3-e44cb8c75b88/exporter/0.log"
Sep 16 19:41:08.978517 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:08.978490    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nrxcw_3900c7ee-b6e7-4edf-84a3-e44cb8c75b88/extractor/0.log"
Sep 16 19:41:11.800051 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:11.800013    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-hx7p6_c0c15d01-2444-4234-b12c-3e032012a314/s3-init/0.log"
Sep 16 19:41:11.818180 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:11.818157    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-hx7p6_c0c15d01-2444-4234-b12c-3e032012a314/download-hf-model/0.log"
Sep 16 19:41:11.849774 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:11.849747    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-2l78c_d3915d78-9c68-4e96-a820-6c7c0fd8a89f/s3-tls-init-custom/0.log"
Sep 16 19:41:11.867450 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:11.867424    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-2l78c_d3915d78-9c68-4e96-a820-6c7c0fd8a89f/download-hf-model/0.log"
Sep 16 19:41:11.899940 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:11.899910    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-9dw49_f7def41f-183c-4d2e-8138-1f5e26837461/s3-tls-init-serving/0.log"
Sep 16 19:41:11.918630 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:11.918604    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-9dw49_f7def41f-183c-4d2e-8138-1f5e26837461/download-hf-model/0.log"
Sep 16 19:41:11.995660 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:11.995613    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-knpk4_a82ff854-4e45-47c5-ba9d-26ce25c223c7/seaweedfs-tls-serving/0.log"
Sep 16 19:41:12.841609 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:12.841584    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t4xfr/perf-node-gather-daemonset-72pc5"
Sep 16 19:41:16.307920 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:16.307883    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-rnfbn_e0a3e579-db97-47c8-bfd2-544ce39f92de/migrator/0.log"
Sep 16 19:41:16.330521 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:16.330437    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-8b4d697c5-rnfbn_e0a3e579-db97-47c8-bfd2-544ce39f92de/graceful-termination/0.log"
Sep 16 19:41:17.682331 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:17.682299    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-frrd7_698bb50e-290e-4842-af73-e8e8f7925f3a/kube-multus-additional-cni-plugins/0.log"
Sep 16 19:41:17.706684 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:17.706655    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-frrd7_698bb50e-290e-4842-af73-e8e8f7925f3a/egress-router-binary-copy/0.log"
Sep 16 19:41:17.729826 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:17.729798    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-frrd7_698bb50e-290e-4842-af73-e8e8f7925f3a/cni-plugins/0.log"
Sep 16 19:41:17.752926 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:17.752894    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-frrd7_698bb50e-290e-4842-af73-e8e8f7925f3a/bond-cni-plugin/0.log"
Sep 16 19:41:17.776153 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:17.776107    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-frrd7_698bb50e-290e-4842-af73-e8e8f7925f3a/routeoverride-cni/0.log"
Sep 16 19:41:17.800294 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:17.800234    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-frrd7_698bb50e-290e-4842-af73-e8e8f7925f3a/whereabouts-cni-bincopy/0.log"
Sep 16 19:41:17.826985 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:17.826955    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-frrd7_698bb50e-290e-4842-af73-e8e8f7925f3a/whereabouts-cni/0.log"
Sep 16 19:41:18.206587 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:18.206509    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-w7rmx_75c1c001-b653-405b-9227-89d064e216bd/kube-multus/0.log"
Sep 16 19:41:18.349592 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:18.349563    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zp95h_d59dba60-7c0d-4e12-b53a-3462d4b9791a/network-metrics-daemon/0.log"
Sep 16 19:41:18.369321 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:18.369297    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-zp95h_d59dba60-7c0d-4e12-b53a-3462d4b9791a/kube-rbac-proxy/0.log"
Sep 16 19:41:19.164299 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:19.164270    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovn-controller/0.log"
Sep 16 19:41:19.182034 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:19.182010    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovn-acl-logging/0.log"
Sep 16 19:41:19.182197 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:19.182010    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovn-acl-logging/1.log"
Sep 16 19:41:19.203047 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:19.203019    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/kube-rbac-proxy-node/0.log"
Sep 16 19:41:19.226353 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:19.226325    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/kube-rbac-proxy-ovn-metrics/0.log"
Sep 16 19:41:19.248332 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:19.248298    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/northd/0.log"
Sep 16 19:41:19.271990 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:19.271955    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/nbdb/0.log"
Sep 16 19:41:19.297691 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:19.297661    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/sbdb/0.log"
Sep 16 19:41:19.347339 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:19.347309    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kdln4_8f7856f8-39ee-46ec-8eb5-48dd0503e543/ovnkube-controller/0.log"
Sep 16 19:41:20.892286 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:20.892254    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-dvtnm_b17c052d-1029-47d7-960f-af161a00f71c/network-check-target-container/0.log"
Sep 16 19:41:21.891842 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:21.891805    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-xr8t8_18c10e7d-2ca0-4852-a696-f3ddebc29f69/iptables-alerter/0.log"
Sep 16 19:41:22.499951 ip-10-0-137-241 kubenswrapper[2554]: I0916 19:41:22.499926    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-6cmnf_c5a1899e-6f2d-4216-aaee-f8281fe6b9dd/tuned/0.log"