Jun 15 05:45:39.759244 ip-10-0-138-16 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jun 15 05:45:39.759256 ip-10-0-138-16 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jun 15 05:45:39.759262 ip-10-0-138-16 systemd[1]: kubelet.service: Failed with result 'resources'.
Jun 15 05:45:39.759508 ip-10-0-138-16 systemd[1]: Failed to start Kubernetes Kubelet.
Jun 15 05:45:49.936592 ip-10-0-138-16 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jun 15 05:45:49.936612 ip-10-0-138-16 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot a9722f486a054d4f8dd214245c21e063 --
Jun 15 05:48:04.724780 ip-10-0-138-16 systemd[1]: Starting Kubernetes Kubelet...
Jun 15 05:48:05.156711 ip-10-0-138-16 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 15 05:48:05.156711 ip-10-0-138-16 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jun 15 05:48:05.156711 ip-10-0-138-16 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 15 05:48:05.156711 ip-10-0-138-16 kubenswrapper[2572]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jun 15 05:48:05.156711 ip-10-0-138-16 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jun 15 05:48:05.158733 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.158643    2572 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jun 15 05:48:05.166590 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166570    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 15 05:48:05.166590 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166586    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 15 05:48:05.166590 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166591    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 15 05:48:05.166590 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166596    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 15 05:48:05.166835 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166600    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 15 05:48:05.166835 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166605    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 15 05:48:05.166835 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166610    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 15 05:48:05.166835 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166614    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 15 05:48:05.166835 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166618    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 15 05:48:05.166835 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166622    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 15 05:48:05.166835 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166626    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 15 05:48:05.166835 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166630    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 15 05:48:05.166835 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166634    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 15 05:48:05.166835 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166638    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jun 15 05:48:05.166835 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166642    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 15 05:48:05.166835 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166645    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jun 15 05:48:05.166835 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166650    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jun 15 05:48:05.166835 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166654    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 15 05:48:05.166835 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166658    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 15 05:48:05.166835 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166662    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 15 05:48:05.166835 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166666    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 15 05:48:05.166835 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166670    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 15 05:48:05.166835 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166673    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 15 05:48:05.166835 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166677    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 15 05:48:05.167673 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166688    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 15 05:48:05.167673 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166692    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 15 05:48:05.167673 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166696    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 15 05:48:05.167673 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166700    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 15 05:48:05.167673 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166704    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jun 15 05:48:05.167673 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166709    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 15 05:48:05.167673 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166714    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 15 05:48:05.167673 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166718    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 15 05:48:05.167673 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166723    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 15 05:48:05.167673 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166727    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jun 15 05:48:05.167673 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166731    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 15 05:48:05.167673 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166735    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jun 15 05:48:05.167673 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166740    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 15 05:48:05.167673 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166744    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 15 05:48:05.167673 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166749    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 15 05:48:05.167673 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166753    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jun 15 05:48:05.167673 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166757    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 15 05:48:05.167673 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166764    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 15 05:48:05.167673 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166771    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 15 05:48:05.168498 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166775    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jun 15 05:48:05.168498 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166780    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 15 05:48:05.168498 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166785    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 15 05:48:05.168498 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166790    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 15 05:48:05.168498 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166796    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jun 15 05:48:05.168498 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166802    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 15 05:48:05.168498 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166807    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jun 15 05:48:05.168498 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166812    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 15 05:48:05.168498 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166816    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 15 05:48:05.168498 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166821    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jun 15 05:48:05.168498 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166825    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 15 05:48:05.168498 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166829    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jun 15 05:48:05.168498 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166834    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jun 15 05:48:05.168498 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166838    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 15 05:48:05.168498 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166853    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 15 05:48:05.168498 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166858    2572 feature_gate.go:328] unrecognized feature gate: Example
Jun 15 05:48:05.168498 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166863    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 15 05:48:05.168498 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166867    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 15 05:48:05.168498 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166871    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 15 05:48:05.169268 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166876    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 15 05:48:05.169268 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166880    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 15 05:48:05.169268 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166885    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 15 05:48:05.169268 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166889    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 15 05:48:05.169268 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166893    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 15 05:48:05.169268 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166897    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 15 05:48:05.169268 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166902    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 15 05:48:05.169268 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166906    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 15 05:48:05.169268 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166911    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 15 05:48:05.169268 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166916    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 15 05:48:05.169268 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166921    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 15 05:48:05.169268 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166925    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 15 05:48:05.169268 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166930    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jun 15 05:48:05.169268 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166934    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 15 05:48:05.169268 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166938    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 15 05:48:05.169268 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166943    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jun 15 05:48:05.169268 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166948    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 15 05:48:05.169268 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166952    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 15 05:48:05.169268 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166958    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 15 05:48:05.169268 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166962    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 15 05:48:05.170131 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166967    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jun 15 05:48:05.170131 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166971    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 15 05:48:05.170131 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166975    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 15 05:48:05.170131 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166979    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 15 05:48:05.170131 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166983    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 15 05:48:05.170131 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166987    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 15 05:48:05.170131 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166992    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 15 05:48:05.170131 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.166997    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 15 05:48:05.170131 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.167001    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 15 05:48:05.170131 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.167006    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 15 05:48:05.170131 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.167010    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 15 05:48:05.170131 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.167016    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jun 15 05:48:05.170131 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.167021    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 15 05:48:05.170131 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.167025    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 15 05:48:05.170131 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.167029    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jun 15 05:48:05.170131 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.167033    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 15 05:48:05.170131 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.167038    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 15 05:48:05.170131 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168245    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 15 05:48:05.170131 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168255    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jun 15 05:48:05.170131 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168263    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jun 15 05:48:05.170999 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168268    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 15 05:48:05.170999 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168273    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 15 05:48:05.170999 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168280    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 15 05:48:05.170999 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168285    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 15 05:48:05.170999 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168289    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 15 05:48:05.170999 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168294    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 15 05:48:05.170999 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168298    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 15 05:48:05.170999 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168302    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 15 05:48:05.170999 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168307    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 15 05:48:05.170999 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168311    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 15 05:48:05.170999 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168316    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 15 05:48:05.170999 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168320    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jun 15 05:48:05.170999 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168324    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 15 05:48:05.170999 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168328    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 15 05:48:05.170999 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168332    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 15 05:48:05.170999 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168337    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 15 05:48:05.170999 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168341    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 15 05:48:05.170999 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168345    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 15 05:48:05.170999 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168350    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 15 05:48:05.170999 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168372    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 15 05:48:05.171789 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168378    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 15 05:48:05.171789 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168382    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jun 15 05:48:05.171789 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168387    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 15 05:48:05.171789 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168394    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 15 05:48:05.171789 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168400    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 15 05:48:05.171789 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168404    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 15 05:48:05.171789 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168408    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jun 15 05:48:05.171789 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168412    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 15 05:48:05.171789 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168416    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jun 15 05:48:05.171789 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168421    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 15 05:48:05.171789 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168425    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 15 05:48:05.171789 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168431    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 15 05:48:05.171789 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168436    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 15 05:48:05.171789 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168440    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jun 15 05:48:05.171789 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168444    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 15 05:48:05.171789 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168449    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 15 05:48:05.171789 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168453    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 15 05:48:05.171789 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168457    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 15 05:48:05.171789 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168461    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 15 05:48:05.171789 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168465    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jun 15 05:48:05.172396 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168469    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 15 05:48:05.172396 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168474    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 15 05:48:05.172396 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168478    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 15 05:48:05.172396 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168482    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 15 05:48:05.172396 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168486    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 15 05:48:05.172396 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168491    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 15 05:48:05.172396 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168494    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 15 05:48:05.172396 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168499    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 15 05:48:05.172396 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168503    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 15 05:48:05.172396 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168507    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 15 05:48:05.172396 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168511    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jun 15 05:48:05.172396 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168515    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 15 05:48:05.172396 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168519    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 15 05:48:05.172396 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168523    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 15 05:48:05.172396 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168527    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 15 05:48:05.172396 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168532    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jun 15 05:48:05.172396 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168537    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 15 05:48:05.172396 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168541    2572 feature_gate.go:328] unrecognized feature gate: Example
Jun 15 05:48:05.172396 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168545    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 15 05:48:05.172396 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168549    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 15 05:48:05.173025 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168554    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jun 15 05:48:05.173025 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168558    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 15 05:48:05.173025 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168562    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jun 15 05:48:05.173025 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168566    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jun 15 05:48:05.173025 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168570    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 15 05:48:05.173025 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168577    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 15 05:48:05.173025 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168582    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jun 15 05:48:05.173025 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168587    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 15 05:48:05.173025 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168592    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 15 05:48:05.173025 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168597    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jun 15 05:48:05.173025 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168601    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 15 05:48:05.173025 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168605    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 15 05:48:05.173025 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168608    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 15 05:48:05.173025 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168613    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 15 05:48:05.173025 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168617    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 15 05:48:05.173025 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168621    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 15 05:48:05.173025 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168627    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 15 05:48:05.173025 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168635    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jun 15 05:48:05.173025 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168640    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 15 05:48:05.173539 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168645    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 15 05:48:05.173539 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168650    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 15 05:48:05.173539 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168654    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 15 05:48:05.173539 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168659    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 15 05:48:05.173539 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168663    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 15 05:48:05.173539 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168668    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 15 05:48:05.173539 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168672    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 15 05:48:05.173539 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168676    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jun 15 05:48:05.173539 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168680    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 15 05:48:05.173539 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168684    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 15 05:48:05.173539 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168688    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 15 05:48:05.173539 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168692    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 15 05:48:05.173539 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168696    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 15 05:48:05.173539 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168700    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 15 05:48:05.173539 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168705    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jun 15 05:48:05.173539 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168709    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 15 05:48:05.173539 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.168713    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 15 05:48:05.173539 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169324    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 15 05:48:05.173539 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169334    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 15 05:48:05.174124 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169339    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jun 15 05:48:05.174124 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169344    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 15 05:48:05.174124 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169351    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 15 05:48:05.174124 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169372    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 15 05:48:05.174124 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169377    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 15 05:48:05.174124 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169381    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 15 05:48:05.174124 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169385    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 15 05:48:05.174124 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169389    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 15 05:48:05.174124 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169393    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 15 05:48:05.174124 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169397    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 15 05:48:05.174124 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169401    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 15 05:48:05.174124 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169406    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jun 15 05:48:05.174124 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169410    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 15 05:48:05.174124 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169415    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 15 05:48:05.174124 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169419    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 15 05:48:05.174124 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169424    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jun 15 05:48:05.174124 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169428    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 15 05:48:05.174124 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169432    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 15 05:48:05.174124 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169436    2572 feature_gate.go:328] unrecognized feature gate: Example
Jun 15 05:48:05.174124 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169440    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jun 15 05:48:05.174642 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169444    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 15 05:48:05.174642 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169449    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 15 05:48:05.174642 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169453    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 15 05:48:05.174642 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169457    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 15 05:48:05.174642 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169461    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 15 05:48:05.174642 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169465    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 15 05:48:05.174642 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169470    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 15 05:48:05.174642 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169474    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 15 05:48:05.174642 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169478    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jun 15 05:48:05.174642 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169482    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 15 05:48:05.174642 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169486    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 15 05:48:05.174642 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169490    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 15 05:48:05.174642 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169496    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 15 05:48:05.174642 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169500    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 15 05:48:05.174642 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169504    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 15 05:48:05.174642 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169510    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jun 15 05:48:05.174642 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169514    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 15 05:48:05.174642 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169518    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 15 05:48:05.174642 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169523    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 15 05:48:05.175175 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169528    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 15 05:48:05.175175 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169533    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 15 05:48:05.175175 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169538    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 15 05:48:05.175175 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169543    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 15 05:48:05.175175 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169547    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 15 05:48:05.175175 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169552    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jun 15 05:48:05.175175 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169556    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 15 05:48:05.175175 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169560    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jun 15 05:48:05.175175 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169564    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 15 05:48:05.175175 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169568    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 15 05:48:05.175175 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169573    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 15 05:48:05.175175 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169577    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 15 05:48:05.175175 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169581    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 15 05:48:05.175175 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169585    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 15 05:48:05.175175 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169590    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 15 05:48:05.175175 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169594    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 15 05:48:05.175175 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169598    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 15 05:48:05.175175 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169602    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jun 15 05:48:05.175175 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169606    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 15 05:48:05.175175 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169610    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 15 05:48:05.175738 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169614    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 15 05:48:05.175738 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169618    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 15 05:48:05.175738 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169622    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 15 05:48:05.175738 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169627    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 15 05:48:05.175738 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169631    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 15 05:48:05.175738 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169635    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 15 05:48:05.175738 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169641    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 15 05:48:05.175738 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169645    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 15 05:48:05.175738 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169649    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 15 05:48:05.175738 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169654    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 15 05:48:05.175738 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169658    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 15 05:48:05.175738 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169662    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 15 05:48:05.175738 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169666    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 15 05:48:05.175738 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169675    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 15 05:48:05.175738 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169679    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jun 15 05:48:05.175738 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169683    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 15 05:48:05.175738 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169687    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 15 05:48:05.175738 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169692    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 15 05:48:05.175738 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169696    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 15 05:48:05.175738 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169701    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 15 05:48:05.176219 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169705    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 15 05:48:05.176219 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169709    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jun 15 05:48:05.176219 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169714    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 15 05:48:05.176219 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169718    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jun 15 05:48:05.176219 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169722    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 15 05:48:05.176219 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169726    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 15 05:48:05.176219 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169733    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jun 15 05:48:05.176219 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169739    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 15 05:48:05.176219 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169744    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 15 05:48:05.176219 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169749    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 15 05:48:05.176219 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169753    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jun 15 05:48:05.176219 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169758    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jun 15 05:48:05.176219 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169763    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jun 15 05:48:05.176219 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169767    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jun 15 05:48:05.176219 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169771    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 15 05:48:05.176219 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169775    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jun 15 05:48:05.176219 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169780    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 15 05:48:05.176219 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.169785    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 15 05:48:05.176219 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.169882    2572 flags.go:64] FLAG: --address="0.0.0.0"
Jun 15 05:48:05.176219 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.169903    2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jun 15 05:48:05.176759 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.169913    2572 flags.go:64] FLAG: --anonymous-auth="true"
Jun 15 05:48:05.176759 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.169920    2572 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jun 15 05:48:05.176759 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.169928    2572 flags.go:64] FLAG: --authentication-token-webhook="false"
Jun 15 05:48:05.176759 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.169933    2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jun 15 05:48:05.176759 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.169940    2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jun 15 05:48:05.176759 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.169949    2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jun 15 05:48:05.176759 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.169954    2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jun 15 05:48:05.176759 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.169959    2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jun 15 05:48:05.176759 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.169965    2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jun 15 05:48:05.176759 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.169970    2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jun 15 05:48:05.176759 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.169975    2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jun 15 05:48:05.176759 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.169981    2572 flags.go:64] FLAG: --cgroup-root=""
Jun 15 05:48:05.176759 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.169985    2572 flags.go:64] FLAG: --cgroups-per-qos="true"
Jun 15 05:48:05.176759 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.169990    2572 flags.go:64] FLAG: --client-ca-file=""
Jun 15 05:48:05.176759 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.169995    2572 flags.go:64] FLAG: --cloud-provider="external"
Jun 15 05:48:05.176759 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170000    2572 flags.go:64] FLAG: --cluster-dns="[]"
Jun 15 05:48:05.176759 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170008    2572 flags.go:64] FLAG: --cluster-domain=""
Jun 15 05:48:05.176759 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170012    2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jun 15 05:48:05.176759 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170018    2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jun 15 05:48:05.176759 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170023    2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jun 15 05:48:05.176759 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170028    2572 flags.go:64] FLAG: --container-log-max-files="5"
Jun 15 05:48:05.176759 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170034    2572 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jun 15 05:48:05.176759 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170039    2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jun 15 05:48:05.177460 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170044    2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jun 15 05:48:05.177460 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170050    2572 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jun 15 05:48:05.177460 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170054    2572 flags.go:64] FLAG: --contention-profiling="false"
Jun 15 05:48:05.177460 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170059    2572 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jun 15 05:48:05.177460 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170064    2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jun 15 05:48:05.177460 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170069    2572 flags.go:64] FLAG: --cpu-manager-policy="none"
Jun 15 05:48:05.177460 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170074    2572 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jun 15 05:48:05.177460 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170081    2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jun 15 05:48:05.177460 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170085    2572 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jun 15 05:48:05.177460 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170100    2572 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jun 15 05:48:05.177460 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170105    2572 flags.go:64] FLAG: --enable-load-reader="false"
Jun 15 05:48:05.177460 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170110    2572 flags.go:64] FLAG: --enable-server="true"
Jun 15 05:48:05.177460 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170115    2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jun 15 05:48:05.177460 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170122    2572 flags.go:64] FLAG: --event-burst="100"
Jun 15 05:48:05.177460 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170129    2572 flags.go:64] FLAG: --event-qps="50"
Jun 15 05:48:05.177460 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170134    2572 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jun 15 05:48:05.177460 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170138    2572 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jun 15 05:48:05.177460 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170143    2572 flags.go:64] FLAG: --eviction-hard=""
Jun 15 05:48:05.177460 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170149    2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jun 15 05:48:05.177460 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170153    2572 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jun 15 05:48:05.177460 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170158    2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jun 15 05:48:05.177460 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170163    2572 flags.go:64] FLAG: --eviction-soft=""
Jun 15 05:48:05.177460 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170168    2572 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jun 15 05:48:05.177460 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170173    2572 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jun 15 05:48:05.177460 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170178    2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jun 15 05:48:05.178102 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170183    2572 flags.go:64] FLAG: --experimental-mounter-path=""
Jun 15 05:48:05.178102 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170187    2572 flags.go:64] FLAG: --fail-cgroupv1="false"
Jun 15 05:48:05.178102 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170192    2572 flags.go:64] FLAG: --fail-swap-on="true"
Jun 15 05:48:05.178102 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170197    2572 flags.go:64] FLAG: --feature-gates=""
Jun 15 05:48:05.178102 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170203    2572 flags.go:64] FLAG: --file-check-frequency="20s"
Jun 15 05:48:05.178102 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170208    2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jun 15 05:48:05.178102 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170213    2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jun 15 05:48:05.178102 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170218    2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jun 15 05:48:05.178102 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170223    2572 flags.go:64] FLAG: --healthz-port="10248"
Jun 15 05:48:05.178102 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170228    2572 flags.go:64] FLAG: --help="false"
Jun 15 05:48:05.178102 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170232    2572 flags.go:64] FLAG: --hostname-override="ip-10-0-138-16.ec2.internal"
Jun 15 05:48:05.178102 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170238    2572 flags.go:64] FLAG: --housekeeping-interval="10s"
Jun 15 05:48:05.178102 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170243    2572 flags.go:64] FLAG: --http-check-frequency="20s"
Jun 15 05:48:05.178102 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170247    2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jun 15 05:48:05.178102 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170253    2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jun 15 05:48:05.178102 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170259    2572 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jun 15 05:48:05.178102 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170265    2572 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jun 15 05:48:05.178102 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170270    2572 flags.go:64] FLAG: --image-service-endpoint=""
Jun 15 05:48:05.178102 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170275    2572 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jun 15 05:48:05.178102 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170279    2572 flags.go:64] FLAG: --kube-api-burst="100"
Jun 15 05:48:05.178102 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170285    2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jun 15 05:48:05.178102 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170291    2572 flags.go:64] FLAG: --kube-api-qps="50"
Jun 15 05:48:05.178102 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170295    2572 flags.go:64] FLAG: --kube-reserved=""
Jun 15 05:48:05.178102 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170300    2572 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jun 15 05:48:05.178712 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170305    2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jun 15 05:48:05.178712 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170309    2572 flags.go:64] FLAG: --kubelet-cgroups=""
Jun 15 05:48:05.178712 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170314    2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jun 15 05:48:05.178712 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170319    2572 flags.go:64] FLAG: --lock-file=""
Jun 15 05:48:05.178712 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170323    2572 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jun 15 05:48:05.178712 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170328    2572 flags.go:64] FLAG: --log-flush-frequency="5s"
Jun 15 05:48:05.178712 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170333    2572 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jun 15 05:48:05.178712 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170342    2572 flags.go:64] FLAG: --log-json-split-stream="false"
Jun 15 05:48:05.178712 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170347    2572 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jun 15 05:48:05.178712 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170351    2572 flags.go:64] FLAG: --log-text-split-stream="false"
Jun 15 05:48:05.178712 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170376    2572 flags.go:64] FLAG: --logging-format="text"
Jun 15 05:48:05.178712 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170382    2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jun 15 05:48:05.178712 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170387    2572 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jun 15 05:48:05.178712 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170392    2572 flags.go:64] FLAG: --manifest-url=""
Jun 15 05:48:05.178712 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170397    2572 flags.go:64] FLAG: --manifest-url-header=""
Jun 15 05:48:05.178712 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170408    2572 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jun 15 05:48:05.178712 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170414    2572 flags.go:64] FLAG: --max-open-files="1000000"
Jun 15 05:48:05.178712 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170420    2572 flags.go:64] FLAG: --max-pods="110"
Jun 15 05:48:05.178712 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170425    2572 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jun 15 05:48:05.178712 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170430    2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jun 15 05:48:05.178712 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170434    2572 flags.go:64] FLAG: --memory-manager-policy="None"
Jun 15 05:48:05.178712 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170439    2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jun 15 05:48:05.178712 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170444    2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jun 15 05:48:05.178712 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170449    2572 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jun 15 05:48:05.178712 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170456    2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jun 15 05:48:05.179329 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170466    2572 flags.go:64] FLAG: --node-status-max-images="50"
Jun 15 05:48:05.179329 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170471    2572 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jun 15 05:48:05.179329 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170476    2572 flags.go:64] FLAG: --oom-score-adj="-999"
Jun 15 05:48:05.179329 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170481    2572 flags.go:64] FLAG: --pod-cidr=""
Jun 15 05:48:05.179329 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170490    2572 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4a2e90c8ef7d783e4d635e24fb09435f5e1f6e954583512c0944f9e57bc77b52"
Jun 15 05:48:05.179329 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170498    2572 flags.go:64] FLAG: --pod-manifest-path=""
Jun 15 05:48:05.179329 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170503    2572 flags.go:64] FLAG: --pod-max-pids="-1"
Jun 15 05:48:05.179329 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170508    2572 flags.go:64] FLAG: --pods-per-core="0"
Jun 15 05:48:05.179329 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170512    2572 flags.go:64] FLAG: --port="10250"
Jun 15 05:48:05.179329 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170517    2572 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jun 15 05:48:05.179329 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170522    2572 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a9dc8dd722ba3e65"
Jun 15 05:48:05.179329 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170527    2572 flags.go:64] FLAG: --qos-reserved=""
Jun 15 05:48:05.179329 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170532    2572 flags.go:64] FLAG: --read-only-port="10255"
Jun 15 05:48:05.179329 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170537    2572 flags.go:64] FLAG: --register-node="true"
Jun 15 05:48:05.179329 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170542    2572 flags.go:64] FLAG: --register-with-taints=""
Jun 15 05:48:05.179329 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170548    2572 flags.go:64] FLAG: --registry-burst="10"
Jun 15 05:48:05.179329 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170553    2572 flags.go:64] FLAG: --registry-qps="5"
Jun 15 05:48:05.179329 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170557    2572 flags.go:64] FLAG: --reserved-cpus=""
Jun 15 05:48:05.179329 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170562    2572 flags.go:64] FLAG: --reserved-memory=""
Jun 15 05:48:05.179329 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170568    2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jun 15 05:48:05.179329 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170574    2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jun 15 05:48:05.179329 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170579    2572 flags.go:64] FLAG: --rotate-certificates="false"
Jun 15 05:48:05.179329 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170583    2572 flags.go:64] FLAG: --rotate-server-certificates="false"
Jun 15 05:48:05.179329 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170588    2572 flags.go:64] FLAG: --runonce="false"
Jun 15 05:48:05.179329 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170592    2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jun 15 05:48:05.179996 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170598    2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jun 15 05:48:05.179996 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170602    2572 flags.go:64] FLAG: --seccomp-default="false"
Jun 15 05:48:05.179996 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170607    2572 flags.go:64] FLAG: --serialize-image-pulls="true"
Jun 15 05:48:05.179996 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170611    2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jun 15 05:48:05.179996 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170616    2572 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jun 15 05:48:05.179996 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170621    2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jun 15 05:48:05.179996 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170628    2572 flags.go:64] FLAG: --storage-driver-password="root"
Jun 15 05:48:05.179996 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170633    2572 flags.go:64] FLAG: --storage-driver-secure="false"
Jun 15 05:48:05.179996 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170638    2572 flags.go:64] FLAG: --storage-driver-table="stats"
Jun 15 05:48:05.179996 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170642    2572 flags.go:64] FLAG: --storage-driver-user="root"
Jun 15 05:48:05.179996 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170647    2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jun 15 05:48:05.179996 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170652    2572 flags.go:64] FLAG: --sync-frequency="1m0s"
Jun 15 05:48:05.179996 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170658    2572 flags.go:64] FLAG: --system-cgroups=""
Jun 15 05:48:05.179996 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170663    2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jun 15 05:48:05.179996 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170672    2572 flags.go:64] FLAG: --system-reserved-cgroup=""
Jun 15 05:48:05.179996 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170676    2572 flags.go:64] FLAG: --tls-cert-file=""
Jun 15 05:48:05.179996 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170681    2572 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jun 15 05:48:05.179996 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170688    2572 flags.go:64] FLAG: --tls-min-version=""
Jun 15 05:48:05.179996 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170693    2572 flags.go:64] FLAG: --tls-private-key-file=""
Jun 15 05:48:05.179996 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170697    2572 flags.go:64] FLAG: --topology-manager-policy="none"
Jun 15 05:48:05.179996 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170702    2572 flags.go:64] FLAG: --topology-manager-policy-options=""
Jun 15 05:48:05.179996 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170707    2572 flags.go:64] FLAG: --topology-manager-scope="container"
Jun 15 05:48:05.179996 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170711    2572 flags.go:64] FLAG: --v="2"
Jun 15 05:48:05.179996 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170717    2572 flags.go:64] FLAG: --version="false"
Jun 15 05:48:05.179996 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170724    2572 flags.go:64] FLAG: --vmodule=""
Jun 15 05:48:05.180650 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170730    2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jun 15 05:48:05.180650 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.170735    2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jun 15 05:48:05.180650 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.170977    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jun 15 05:48:05.180650 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.170984    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 15 05:48:05.180650 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.170989    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 15 05:48:05.180650 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.170994    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jun 15 05:48:05.180650 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.170998    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jun 15 05:48:05.180650 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171003    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 15 05:48:05.180650 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171007    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 15 05:48:05.180650 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171011    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 15 05:48:05.180650 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171016    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 15 05:48:05.180650 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171021    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 15 05:48:05.180650 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171026    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jun 15 05:48:05.180650 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171033    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 15 05:48:05.180650 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171037    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 15 05:48:05.180650 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171041    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 15 05:48:05.180650 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171045    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jun 15 05:48:05.180650 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171052    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jun 15 05:48:05.180650 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171057    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 15 05:48:05.180650 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171062    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 15 05:48:05.181145 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171067    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 15 05:48:05.181145 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171072    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 15 05:48:05.181145 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171077    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 15 05:48:05.181145 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171081    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 15 05:48:05.181145 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171085    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 15 05:48:05.181145 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171090    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 15 05:48:05.181145 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171094    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 15 05:48:05.181145 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171100    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 15 05:48:05.181145 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171104    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jun 15 05:48:05.181145 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171108    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 15 05:48:05.181145 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171113    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 15 05:48:05.181145 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171117    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 15 05:48:05.181145 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171121    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 15 05:48:05.181145 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171126    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 15 05:48:05.181145 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171130    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 15 05:48:05.181145 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171134    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 15 05:48:05.181145 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171139    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 15 05:48:05.181145 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171143    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 15 05:48:05.181145 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171147    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 15 05:48:05.181145 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171152    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 15 05:48:05.181656 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171156    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jun 15 05:48:05.181656 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171160    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jun 15 05:48:05.181656 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171164    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 15 05:48:05.181656 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171168    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 15 05:48:05.181656 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171175    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 15 05:48:05.181656 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171182    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 15 05:48:05.181656 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171187    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jun 15 05:48:05.181656 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171191    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 15 05:48:05.181656 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171195    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 15 05:48:05.181656 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171199    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 15 05:48:05.181656 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171204    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jun 15 05:48:05.181656 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171210    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jun 15 05:48:05.181656 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171214    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 15 05:48:05.181656 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171219    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 15 05:48:05.181656 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171223    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 15 05:48:05.181656 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171227    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 15 05:48:05.181656 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171232    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jun 15 05:48:05.181656 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171236    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jun 15 05:48:05.181656 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171240    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 15 05:48:05.182132 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171244    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 15 05:48:05.182132 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171248    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 15 05:48:05.182132 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171252    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 15 05:48:05.182132 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171256    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 15 05:48:05.182132 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171261    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 15 05:48:05.182132 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171265    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 15 05:48:05.182132 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171270    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 15 05:48:05.182132 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171274    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 15 05:48:05.182132 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171279    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 15 05:48:05.182132 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171283    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 15 05:48:05.182132 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171287    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 15 05:48:05.182132 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171291    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 15 05:48:05.182132 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171296    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 15 05:48:05.182132 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171300    2572 feature_gate.go:328] unrecognized feature gate: Example
Jun 15 05:48:05.182132 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171304    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 15 05:48:05.182132 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171308    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jun 15 05:48:05.182132 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171312    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 15 05:48:05.182132 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171316    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 15 05:48:05.182132 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171323    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 15 05:48:05.182132 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171327    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 15 05:48:05.182634 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171331    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 15 05:48:05.182634 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171336    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 15 05:48:05.182634 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171340    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 15 05:48:05.182634 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171344    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 15 05:48:05.182634 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171350    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 15 05:48:05.182634 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171372    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 15 05:48:05.182634 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171377    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 15 05:48:05.182634 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171382    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 15 05:48:05.182634 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171386    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 15 05:48:05.182634 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171390    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jun 15 05:48:05.182634 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171397    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 15 05:48:05.182634 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171403    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 15 05:48:05.182634 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171407    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 15 05:48:05.182634 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171411    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 15 05:48:05.182634 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171415    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 15 05:48:05.182634 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171419    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jun 15 05:48:05.182634 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171424    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jun 15 05:48:05.182634 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171428    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 15 05:48:05.182634 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171432    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 15 05:48:05.183097 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171437    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 15 05:48:05.183097 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171441    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 15 05:48:05.183097 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.171445    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 15 05:48:05.183097 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.171453    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jun 15 05:48:05.183097 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.178412    2572 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jun 15 05:48:05.183097 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.178518    2572 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jun 15 05:48:05.183097 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.178538    2572 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jun 15 05:48:05.183097 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.178544    2572 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jun 15 05:48:05.183097 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178603    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 15 05:48:05.183097 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178608    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jun 15 05:48:05.183097 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178611    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jun 15 05:48:05.183097 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178615    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 15 05:48:05.183097 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178618    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 15 05:48:05.183097 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178621    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jun 15 05:48:05.183097 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178624    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 15 05:48:05.183488 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178627    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 15 05:48:05.183488 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178630    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 15 05:48:05.183488 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178632    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 15 05:48:05.183488 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178635    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jun 15 05:48:05.183488 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178638    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 15 05:48:05.183488 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178640    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 15 05:48:05.183488 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178643    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 15 05:48:05.183488 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178646    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 15 05:48:05.183488 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178649    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 15 05:48:05.183488 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178651    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 15 05:48:05.183488 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178654    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 15 05:48:05.183488 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178656    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 15 05:48:05.183488 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178659    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jun 15 05:48:05.183488 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178661    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jun 15 05:48:05.183488 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178664    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 15 05:48:05.183488 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178667    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 15 05:48:05.183488 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178669    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jun 15 05:48:05.183488 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178672    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 15 05:48:05.183488 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178674    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 15 05:48:05.183488 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178677    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jun 15 05:48:05.183962 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178680    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 15 05:48:05.183962 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178683    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jun 15 05:48:05.183962 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178688    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 15 05:48:05.183962 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178691    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 15 05:48:05.183962 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178695    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 15 05:48:05.183962 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178698    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 15 05:48:05.183962 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178701    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jun 15 05:48:05.183962 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178703    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 15 05:48:05.183962 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178706    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 15 05:48:05.183962 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178708    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 15 05:48:05.183962 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178711    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 15 05:48:05.183962 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178713    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 15 05:48:05.183962 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178716    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 15 05:48:05.183962 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178719    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 15 05:48:05.183962 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178722    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 15 05:48:05.183962 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178724    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 15 05:48:05.183962 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178727    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jun 15 05:48:05.183962 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178730    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 15 05:48:05.183962 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178732    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 15 05:48:05.184460 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178735    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 15 05:48:05.184460 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178738    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jun 15 05:48:05.184460 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178740    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 15 05:48:05.184460 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178743    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 15 05:48:05.184460 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178745    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 15 05:48:05.184460 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178748    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 15 05:48:05.184460 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178751    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 15 05:48:05.184460 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178753    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 15 05:48:05.184460 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178756    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 15 05:48:05.184460 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178758    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 15 05:48:05.184460 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178761    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 15 05:48:05.184460 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178763    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 15 05:48:05.184460 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178766    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 15 05:48:05.184460 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178769    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 15 05:48:05.184460 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178771    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 15 05:48:05.184460 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178774    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jun 15 05:48:05.184460 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178778    2572 feature_gate.go:328] unrecognized feature gate: Example
Jun 15 05:48:05.184460 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178780    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 15 05:48:05.184460 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178783    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 15 05:48:05.184460 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178785    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 15 05:48:05.184949 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178788    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 15 05:48:05.184949 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178790    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 15 05:48:05.184949 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178793    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 15 05:48:05.184949 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178796    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 15 05:48:05.184949 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178798    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 15 05:48:05.184949 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178800    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jun 15 05:48:05.184949 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178803    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 15 05:48:05.184949 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178806    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 15 05:48:05.184949 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178810    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 15 05:48:05.184949 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178813    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 15 05:48:05.184949 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178816    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 15 05:48:05.184949 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178818    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 15 05:48:05.184949 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178821    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 15 05:48:05.184949 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178824    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 15 05:48:05.184949 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178826    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 15 05:48:05.184949 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178829    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 15 05:48:05.184949 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178831    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 15 05:48:05.184949 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178833    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 15 05:48:05.184949 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178836    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jun 15 05:48:05.184949 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178839    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 15 05:48:05.185450 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178841    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 15 05:48:05.185450 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178843    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 15 05:48:05.185450 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178847    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jun 15 05:48:05.185450 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178851    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 15 05:48:05.185450 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178854    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jun 15 05:48:05.185450 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178857    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jun 15 05:48:05.185450 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178860    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 15 05:48:05.185450 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178864    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 15 05:48:05.185450 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178867    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 15 05:48:05.185450 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178870    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 15 05:48:05.185450 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178873    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 15 05:48:05.185450 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178875    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 15 05:48:05.185450 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.178878    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 15 05:48:05.185450 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.178883    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jun 15 05:48:05.185450 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179002    2572 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jun 15 05:48:05.185852 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179006    2572 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jun 15 05:48:05.185852 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179009    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jun 15 05:48:05.185852 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179012    2572 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jun 15 05:48:05.185852 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179014    2572 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jun 15 05:48:05.185852 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179017    2572 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jun 15 05:48:05.185852 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179020    2572 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jun 15 05:48:05.185852 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179022    2572 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jun 15 05:48:05.185852 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179024    2572 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jun 15 05:48:05.185852 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179028    2572 feature_gate.go:328] unrecognized feature gate: Example
Jun 15 05:48:05.185852 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179030    2572 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jun 15 05:48:05.185852 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179033    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jun 15 05:48:05.185852 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179035    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jun 15 05:48:05.185852 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179038    2572 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jun 15 05:48:05.185852 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179040    2572 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jun 15 05:48:05.185852 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179043    2572 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jun 15 05:48:05.185852 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179045    2572 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jun 15 05:48:05.185852 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179048    2572 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jun 15 05:48:05.185852 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179050    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jun 15 05:48:05.185852 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179053    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jun 15 05:48:05.185852 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179055    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jun 15 05:48:05.186338 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179058    2572 feature_gate.go:328] unrecognized feature gate: Example2
Jun 15 05:48:05.186338 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179061    2572 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jun 15 05:48:05.186338 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179063    2572 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jun 15 05:48:05.186338 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179066    2572 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jun 15 05:48:05.186338 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179069    2572 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jun 15 05:48:05.186338 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179071    2572 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jun 15 05:48:05.186338 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179074    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jun 15 05:48:05.186338 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179081    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jun 15 05:48:05.186338 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179084    2572 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jun 15 05:48:05.186338 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179087    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jun 15 05:48:05.186338 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179089    2572 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jun 15 05:48:05.186338 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179091    2572 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jun 15 05:48:05.186338 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179094    2572 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jun 15 05:48:05.186338 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179097    2572 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jun 15 05:48:05.186338 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179099    2572 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jun 15 05:48:05.186338 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179101    2572 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jun 15 05:48:05.186338 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179104    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jun 15 05:48:05.186338 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179106    2572 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jun 15 05:48:05.186338 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179109    2572 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jun 15 05:48:05.186338 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179112    2572 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jun 15 05:48:05.186864 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179114    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jun 15 05:48:05.186864 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179117    2572 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jun 15 05:48:05.186864 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179119    2572 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jun 15 05:48:05.186864 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179122    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jun 15 05:48:05.186864 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179124    2572 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jun 15 05:48:05.186864 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179127    2572 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jun 15 05:48:05.186864 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179130    2572 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jun 15 05:48:05.186864 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179132    2572 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jun 15 05:48:05.186864 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179134    2572 feature_gate.go:328] unrecognized feature gate: EventTTL
Jun 15 05:48:05.186864 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179138    2572 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jun 15 05:48:05.186864 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179142    2572 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jun 15 05:48:05.186864 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179146    2572 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jun 15 05:48:05.186864 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179149    2572 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jun 15 05:48:05.186864 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179151    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jun 15 05:48:05.186864 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179154    2572 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jun 15 05:48:05.186864 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179157    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jun 15 05:48:05.186864 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179160    2572 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jun 15 05:48:05.186864 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179163    2572 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jun 15 05:48:05.186864 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179165    2572 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jun 15 05:48:05.187446 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179167    2572 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jun 15 05:48:05.187446 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179171    2572 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jun 15 05:48:05.187446 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179173    2572 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jun 15 05:48:05.187446 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179176    2572 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jun 15 05:48:05.187446 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179178    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jun 15 05:48:05.187446 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179180    2572 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jun 15 05:48:05.187446 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179183    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jun 15 05:48:05.187446 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179186    2572 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jun 15 05:48:05.187446 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179189    2572 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jun 15 05:48:05.187446 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179193    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jun 15 05:48:05.187446 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179196    2572 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jun 15 05:48:05.187446 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179199    2572 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jun 15 05:48:05.187446 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179201    2572 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jun 15 05:48:05.187446 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179204    2572 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jun 15 05:48:05.187446 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179207    2572 feature_gate.go:328] unrecognized feature gate: OSStreams
Jun 15 05:48:05.187446 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179210    2572 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jun 15 05:48:05.187446 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179213    2572 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jun 15 05:48:05.187446 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179215    2572 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jun 15 05:48:05.187446 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179217    2572 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jun 15 05:48:05.187446 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179220    2572 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jun 15 05:48:05.188283 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179223    2572 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jun 15 05:48:05.188283 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179225    2572 feature_gate.go:328] unrecognized feature gate: NewOLM
Jun 15 05:48:05.188283 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179227    2572 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jun 15 05:48:05.188283 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179230    2572 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jun 15 05:48:05.188283 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179232    2572 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jun 15 05:48:05.188283 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179235    2572 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jun 15 05:48:05.188283 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179237    2572 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jun 15 05:48:05.188283 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179240    2572 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jun 15 05:48:05.188283 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179242    2572 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jun 15 05:48:05.188283 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179245    2572 feature_gate.go:328] unrecognized feature gate: DualReplica
Jun 15 05:48:05.188283 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179248    2572 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jun 15 05:48:05.188283 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179250    2572 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jun 15 05:48:05.188283 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179253    2572 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jun 15 05:48:05.188283 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179255    2572 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jun 15 05:48:05.188283 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179258    2572 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jun 15 05:48:05.188283 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179261    2572 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jun 15 05:48:05.188283 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179263    2572 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jun 15 05:48:05.188283 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179266    2572 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jun 15 05:48:05.188283 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:05.179268    2572 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jun 15 05:48:05.188890 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.179273    2572 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jun 15 05:48:05.188890 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.179400    2572 server.go:962] "Client rotation is on, will bootstrap in background"
Jun 15 05:48:05.188890 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.181642    2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jun 15 05:48:05.188890 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.183235    2572 server.go:1019] "Starting client certificate rotation"
Jun 15 05:48:05.188890 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.183332    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jun 15 05:48:05.188890 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.183373    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jun 15 05:48:05.188890 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.185983    2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jun 15 05:48:05.188890 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.187796    2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jun 15 05:48:05.200117 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.200096    2572 log.go:25] "Validated CRI v1 runtime API"
Jun 15 05:48:05.204057 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.204043    2572 log.go:25] "Validated CRI v1 image API"
Jun 15 05:48:05.204854 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.204843    2572 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jun 15 05:48:05.210860 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.210839    2572 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8d648b03-64e2-425d-beff-4ac5231a6c6e:/dev/nvme0n1p3 cbabd736-e03c-47d0-ba46-a1e8c0992415:/dev/nvme0n1p4]
Jun 15 05:48:05.210948 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.210858    2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jun 15 05:48:05.212823 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.212807    2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jun 15 05:48:05.215999 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.215887    2572 manager.go:216] Machine: {Timestamp:2026-06-15 05:48:05.214480928 +0000 UTC m=+0.362660920 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100564 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28d7a141933ea48cee37b2614305fa SystemUUID:ec28d7a1-4193-3ea4-8cee-37b2614305fa BootID:a9722f48-6a05-4d4f-8dd2-14245c21e063 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8c:b5:93:e6:dd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8c:b5:93:e6:dd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ba:9c:c5:33:03:02 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jun 15 05:48:05.215999 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.215995    2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jun 15 05:48:05.216111 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.216100    2572 manager.go:232] Version: {KernelVersion:5.14.0-570.120.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260608-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jun 15 05:48:05.218111 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.218084    2572 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jun 15 05:48:05.218249 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.218113    2572 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-16.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jun 15 05:48:05.219610 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.219599    2572 topology_manager.go:138] "Creating topology manager with none policy"
Jun 15 05:48:05.219642 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.219613    2572 container_manager_linux.go:309] "Creating device plugin manager"
Jun 15 05:48:05.219642 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.219621    2572 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jun 15 05:48:05.220373 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.220353    2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jun 15 05:48:05.222047 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.222038    2572 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jun 15 05:48:05.226229 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.226218    2572 state_mem.go:36] "Initialized new in-memory state store"
Jun 15 05:48:05.226336 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.226327    2572 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jun 15 05:48:05.226386 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.226374    2572 kubelet.go:486] "Attempting to sync node with API server"
Jun 15 05:48:05.226386 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.226384    2572 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jun 15 05:48:05.226492 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.226402    2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jun 15 05:48:05.226492 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.226410    2572 kubelet.go:398] "Adding apiserver pod source"
Jun 15 05:48:05.226492 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.226419    2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jun 15 05:48:05.226754 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.226738    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jun 15 05:48:05.226810 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.226758    2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jun 15 05:48:05.229172 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.229154    2572 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jun 15 05:48:05.230873 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.230856    2572 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jun 15 05:48:05.230963 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.230895    2572 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jun 15 05:48:05.231097 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.231084    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jun 15 05:48:05.231155 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.231104    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jun 15 05:48:05.231155 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.231114    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jun 15 05:48:05.231155 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.231123    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jun 15 05:48:05.231155 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.231136    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jun 15 05:48:05.231155 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.231146    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jun 15 05:48:05.231718 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.231707    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jun 15 05:48:05.231772 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.231720    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jun 15 05:48:05.231772 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.231732    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jun 15 05:48:05.231772 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.231741    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jun 15 05:48:05.231772 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.231754    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jun 15 05:48:05.231772 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.231766    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jun 15 05:48:05.232649 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.232638    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jun 15 05:48:05.232713 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.232651    2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jun 15 05:48:05.235738 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.235722    2572 server.go:1268] "Started kubelet"
Jun 15 05:48:05.235858 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.235781    2572 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jun 15 05:48:05.235980 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.235835    2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jun 15 05:48:05.235980 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.235903    2572 server_v1.go:49] "podresources" method="list" useActivePods=true
Jun 15 05:48:05.236738 ip-10-0-138-16 systemd[1]: Started Kubernetes Kubelet.
Jun 15 05:48:05.240236 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.240207    2572 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jun 15 05:48:05.240386 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.240369    2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-16.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jun 15 05:48:05.240552 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:05.239931    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-16.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jun 15 05:48:05.240719 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:05.240696    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jun 15 05:48:05.242101 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.242077    2572 server.go:310] "Adding debug handlers to kubelet server"
Jun 15 05:48:05.246035 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.246007    2572 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jun 15 05:48:05.246174 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.246124    2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jun 15 05:48:05.246297 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:05.246275    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-16.ec2.internal\" not found"
Jun 15 05:48:05.246984 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.246964    2572 volume_manager.go:311] "The desired_state_of_world populator starts"
Jun 15 05:48:05.246984 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.246987    2572 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jun 15 05:48:05.247120 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.247012    2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jun 15 05:48:05.247120 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.247097    2572 reconstruct.go:97] "Volume reconstruction finished"
Jun 15 05:48:05.247120 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.247104    2572 reconciler.go:29] "Reconciler: start to sync state"
Jun 15 05:48:05.247120 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:05.247110    2572 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jun 15 05:48:05.247399 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:05.245889    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-16.ec2.internal.18b92b124ef3012f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-16.ec2.internal,UID:ip-10-0-138-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-16.ec2.internal,},FirstTimestamp:2026-06-15 05:48:05.235695919 +0000 UTC m=+0.383875906,LastTimestamp:2026-06-15 05:48:05.235695919 +0000 UTC m=+0.383875906,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-16.ec2.internal,}"
Jun 15 05:48:05.247796 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.247776    2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jun 15 05:48:05.247918 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.247890    2572 factory.go:55] Registering systemd factory
Jun 15 05:48:05.247918 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.247911    2572 factory.go:223] Registration of the systemd container factory successfully
Jun 15 05:48:05.248137 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.248125    2572 factory.go:153] Registering CRI-O factory
Jun 15 05:48:05.248173 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.248139    2572 factory.go:223] Registration of the crio container factory successfully
Jun 15 05:48:05.248173 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.248168    2572 factory.go:103] Registering Raw factory
Jun 15 05:48:05.248243 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.248183    2572 manager.go:1196] Started watching for new ooms in manager
Jun 15 05:48:05.248653 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.248639    2572 manager.go:349] Starting recovery of all containers
Jun 15 05:48:05.253896 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:05.253872    2572 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-16.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jun 15 05:48:05.253973 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:05.253930    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jun 15 05:48:05.259224 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.259209    2572 manager.go:354] Recovery completed
Jun 15 05:48:05.262989 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.262973    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jun 15 05:48:05.266528 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.266503    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-16.ec2.internal" event="NodeHasSufficientMemory"
Jun 15 05:48:05.266604 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.266545    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-16.ec2.internal" event="NodeHasNoDiskPressure"
Jun 15 05:48:05.266604 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.266561    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-16.ec2.internal" event="NodeHasSufficientPID"
Jun 15 05:48:05.267052 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.267038    2572 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jun 15 05:48:05.267052 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.267050    2572 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jun 15 05:48:05.267142 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.267067    2572 state_mem.go:36] "Initialized new in-memory state store"
Jun 15 05:48:05.267142 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.267093    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jun 15 05:48:05.268522 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.268504    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-16.ec2.internal" event="NodeHasSufficientMemory"
Jun 15 05:48:05.268592 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.268543    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-16.ec2.internal" event="NodeHasNoDiskPressure"
Jun 15 05:48:05.268592 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.268558    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-16.ec2.internal" event="NodeHasSufficientPID"
Jun 15 05:48:05.268743 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:05.268682    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-16.ec2.internal.18b92b1250c9771f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-16.ec2.internal,UID:ip-10-0-138-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-16.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-16.ec2.internal,},FirstTimestamp:2026-06-15 05:48:05.266528031 +0000 UTC m=+0.414708025,LastTimestamp:2026-06-15 05:48:05.266528031 +0000 UTC m=+0.414708025,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-16.ec2.internal,}"
Jun 15 05:48:05.269147 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.269134    2572 policy_none.go:49] "None policy: Start"
Jun 15 05:48:05.269211 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.269152    2572 memory_manager.go:187] "Starting memorymanager" policy="None"
Jun 15 05:48:05.269211 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.269162    2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jun 15 05:48:05.271025 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.271012    2572 policy_none.go:47] "Start"
Jun 15 05:48:05.277700 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:05.277635    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-16.ec2.internal.18b92b1250c9d5bf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-16.ec2.internal,UID:ip-10-0-138-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-16.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-16.ec2.internal,},FirstTimestamp:2026-06-15 05:48:05.266552255 +0000 UTC m=+0.414732246,LastTimestamp:2026-06-15 05:48:05.266552255 +0000 UTC m=+0.414732246,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-16.ec2.internal,}"
Jun 15 05:48:05.287976 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:05.287916    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-16.ec2.internal.18b92b1250ca0fe3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-16.ec2.internal,UID:ip-10-0-138-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-16.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-16.ec2.internal,},FirstTimestamp:2026-06-15 05:48:05.266567139 +0000 UTC m=+0.414747134,LastTimestamp:2026-06-15 05:48:05.266567139 +0000 UTC m=+0.414747134,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-16.ec2.internal,}"
Jun 15 05:48:05.298655 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:05.298596    2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-16.ec2.internal.18b92b1250c9771f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-16.ec2.internal.18b92b1250c9771f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-16.ec2.internal,UID:ip-10-0-138-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-16.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-16.ec2.internal,},FirstTimestamp:2026-06-15 05:48:05.266528031 +0000 UTC m=+0.414708025,LastTimestamp:2026-06-15 05:48:05.268519636 +0000 UTC m=+0.416699630,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-16.ec2.internal,}"
Jun 15 05:48:05.307880 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:05.307810    2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-16.ec2.internal.18b92b1250c9d5bf\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-16.ec2.internal.18b92b1250c9d5bf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-16.ec2.internal,UID:ip-10-0-138-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-16.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-16.ec2.internal,},FirstTimestamp:2026-06-15 05:48:05.266552255 +0000 UTC m=+0.414732246,LastTimestamp:2026-06-15 05:48:05.26855073 +0000 UTC m=+0.416730718,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-16.ec2.internal,}"
Jun 15 05:48:05.332592 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.310914    2572 manager.go:337] "Starting Device Plugin manager"
Jun 15 05:48:05.332592 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:05.310970    2572 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jun 15 05:48:05.332592 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.310979    2572 server.go:87] "Starting device plugin registration server"
Jun 15 05:48:05.332592 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.311200    2572 eviction_manager.go:189] "Eviction manager: starting control loop"
Jun 15 05:48:05.332592 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.311209    2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jun 15 05:48:05.332592 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.311330    2572 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jun 15 05:48:05.332592 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.311428    2572 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jun 15 05:48:05.332592 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.311439    2572 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jun 15 05:48:05.332592 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:05.312187    2572 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jun 15 05:48:05.332592 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:05.312223    2572 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-16.ec2.internal\" not found"
Jun 15 05:48:05.332592 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:05.315165    2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-16.ec2.internal.18b92b1250ca0fe3\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-16.ec2.internal.18b92b1250ca0fe3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-16.ec2.internal,UID:ip-10-0-138-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-16.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-16.ec2.internal,},FirstTimestamp:2026-06-15 05:48:05.266567139 +0000 UTC m=+0.414747134,LastTimestamp:2026-06-15 05:48:05.268563682 +0000 UTC m=+0.416743674,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-16.ec2.internal,}"
Jun 15 05:48:05.332592 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.316810    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jun 15 05:48:05.332592 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.317933    2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jun 15 05:48:05.332592 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.317963    2572 status_manager.go:244] "Starting to sync pod status with apiserver"
Jun 15 05:48:05.332592 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.317980    2572 kubelet.go:2448] "Starting kubelet main sync loop"
Jun 15 05:48:05.332592 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:05.318043    2572 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jun 15 05:48:05.332592 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:05.327909    2572 reflector.go:205] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass"
Jun 15 05:48:05.332592 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:05.327951    2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-16.ec2.internal.18b92b1253f6db44  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-16.ec2.internal,UID:ip-10-0-138-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-138-16.ec2.internal,},FirstTimestamp:2026-06-15 05:48:05.319834436 +0000 UTC m=+0.468014411,LastTimestamp:2026-06-15 05:48:05.319834436 +0000 UTC m=+0.468014411,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-16.ec2.internal,}"
Jun 15 05:48:05.338433 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.338416    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jznz4"
Jun 15 05:48:05.344605 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.344589    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-jznz4"
Jun 15 05:48:05.413137 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.413077    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jun 15 05:48:05.414554 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.414539    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-16.ec2.internal" event="NodeHasSufficientMemory"
Jun 15 05:48:05.414631 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.414570    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-16.ec2.internal" event="NodeHasNoDiskPressure"
Jun 15 05:48:05.414631 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.414582    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-16.ec2.internal" event="NodeHasSufficientPID"
Jun 15 05:48:05.414631 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.414606    2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-16.ec2.internal"
Jun 15 05:48:05.417402 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:05.417320    2572 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-138-16.ec2.internal.18b92b1250c9771f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-16.ec2.internal.18b92b1250c9771f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-16.ec2.internal,UID:ip-10-0-138-16.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-16.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-16.ec2.internal,},FirstTimestamp:2026-06-15 05:48:05.266528031 +0000 UTC m=+0.414708025,LastTimestamp:2026-06-15 05:48:05.414554606 +0000 UTC m=+0.562734594,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-16.ec2.internal,}"
Jun 15 05:48:05.418917 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.418901    2572 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-16.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-16.ec2.internal"]
Jun 15 05:48:05.419010 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.418961    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jun 15 05:48:05.419732 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.419719    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-16.ec2.internal" event="NodeHasSufficientMemory"
Jun 15 05:48:05.419791 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.419743    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-16.ec2.internal" event="NodeHasNoDiskPressure"
Jun 15 05:48:05.419791 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.419752    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-16.ec2.internal" event="NodeHasSufficientPID"
Jun 15 05:48:05.421910 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.421895    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jun 15 05:48:05.422012 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.421997    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-16.ec2.internal"
Jun 15 05:48:05.422070 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.422060    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jun 15 05:48:05.422915 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.422896    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-16.ec2.internal" event="NodeHasSufficientMemory"
Jun 15 05:48:05.423031 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.422923    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-16.ec2.internal" event="NodeHasNoDiskPressure"
Jun 15 05:48:05.423031 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.422949    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-16.ec2.internal" event="NodeHasSufficientPID"
Jun 15 05:48:05.423031 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.422952    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-16.ec2.internal" event="NodeHasSufficientMemory"
Jun 15 05:48:05.423031 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.422971    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-16.ec2.internal" event="NodeHasNoDiskPressure"
Jun 15 05:48:05.423031 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.422987    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-16.ec2.internal" event="NodeHasSufficientPID"
Jun 15 05:48:05.425192 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.425176    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-16.ec2.internal"
Jun 15 05:48:05.425248 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.425208    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jun 15 05:48:05.425891 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.425875    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-16.ec2.internal" event="NodeHasSufficientMemory"
Jun 15 05:48:05.425966 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.425898    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-16.ec2.internal" event="NodeHasNoDiskPressure"
Jun 15 05:48:05.425966 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.425908    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-16.ec2.internal" event="NodeHasSufficientPID"
Jun 15 05:48:05.431200 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:05.431184    2572 kubelet_node_status.go:116] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-138-16.ec2.internal\" not found" node="ip-10-0-138-16.ec2.internal"
Jun 15 05:48:05.449039 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.449018    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/22301900658e933839cf55f1e43739b1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-16.ec2.internal\" (UID: \"22301900658e933839cf55f1e43739b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-16.ec2.internal"
Jun 15 05:48:05.449137 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.449050    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22301900658e933839cf55f1e43739b1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-16.ec2.internal\" (UID: \"22301900658e933839cf55f1e43739b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-16.ec2.internal"
Jun 15 05:48:05.449137 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.449066    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f55c91d0f7b296b874c42e3338400df2-config\") pod \"kube-apiserver-proxy-ip-10-0-138-16.ec2.internal\" (UID: \"f55c91d0f7b296b874c42e3338400df2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-16.ec2.internal"
Jun 15 05:48:05.449137 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:05.449126    2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-16.ec2.internal\" not found" node="ip-10-0-138-16.ec2.internal"
Jun 15 05:48:05.453494 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:05.453479    2572 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-16.ec2.internal\" not found" node="ip-10-0-138-16.ec2.internal"
Jun 15 05:48:05.460312 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:05.460298    2572 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-138-16.ec2.internal\" not found" node="ip-10-0-138-16.ec2.internal"
Jun 15 05:48:05.550167 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.550143    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/22301900658e933839cf55f1e43739b1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-16.ec2.internal\" (UID: \"22301900658e933839cf55f1e43739b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-16.ec2.internal"
Jun 15 05:48:05.550274 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.550169    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22301900658e933839cf55f1e43739b1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-16.ec2.internal\" (UID: \"22301900658e933839cf55f1e43739b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-16.ec2.internal"
Jun 15 05:48:05.550274 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.550187    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f55c91d0f7b296b874c42e3338400df2-config\") pod \"kube-apiserver-proxy-ip-10-0-138-16.ec2.internal\" (UID: \"f55c91d0f7b296b874c42e3338400df2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-16.ec2.internal"
Jun 15 05:48:05.550274 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.550230    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f55c91d0f7b296b874c42e3338400df2-config\") pod \"kube-apiserver-proxy-ip-10-0-138-16.ec2.internal\" (UID: \"f55c91d0f7b296b874c42e3338400df2\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-16.ec2.internal"
Jun 15 05:48:05.550274 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.550241    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/22301900658e933839cf55f1e43739b1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-16.ec2.internal\" (UID: \"22301900658e933839cf55f1e43739b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-16.ec2.internal"
Jun 15 05:48:05.550274 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.550264    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22301900658e933839cf55f1e43739b1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-16.ec2.internal\" (UID: \"22301900658e933839cf55f1e43739b1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-16.ec2.internal"
Jun 15 05:48:05.631289 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.631241    2572 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jun 15 05:48:05.632204 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.632187    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-16.ec2.internal" event="NodeHasSufficientMemory"
Jun 15 05:48:05.632277 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.632220    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-16.ec2.internal" event="NodeHasNoDiskPressure"
Jun 15 05:48:05.632277 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.632232    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-16.ec2.internal" event="NodeHasSufficientPID"
Jun 15 05:48:05.632277 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.632265    2572 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-16.ec2.internal"
Jun 15 05:48:05.641126 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.641108    2572 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-16.ec2.internal"
Jun 15 05:48:05.641233 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:05.641135    2572 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-16.ec2.internal\": node \"ip-10-0-138-16.ec2.internal\" not found"
Jun 15 05:48:05.655714 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:05.655699    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-16.ec2.internal\" not found"
Jun 15 05:48:05.752421 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.752320    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-16.ec2.internal"
Jun 15 05:48:05.756560 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:05.756314    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-16.ec2.internal\" not found"
Jun 15 05:48:05.758482 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:05.758468    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-16.ec2.internal"
Jun 15 05:48:05.857015 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:05.856975    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-16.ec2.internal\" not found"
Jun 15 05:48:05.957484 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:05.957452    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-16.ec2.internal\" not found"
Jun 15 05:48:06.058063 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:06.057983    2572 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-16.ec2.internal\" not found"
Jun 15 05:48:06.097309 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.097290    2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 15 05:48:06.156965 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.156942    2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 15 05:48:06.182699 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.182672    2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jun 15 05:48:06.182854 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.182810    2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jun 15 05:48:06.182910 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.182849    2572 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jun 15 05:48:06.227025 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.226993    2572 apiserver.go:52] "Watching apiserver"
Jun 15 05:48:06.244609 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.244583    2572 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jun 15 05:48:06.244956 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.244931    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-z9ws7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j6vx","openshift-dns/node-resolver-4dv8m","openshift-multus/multus-additional-cni-plugins-jwgfl","openshift-multus/multus-ccxxl","openshift-multus/network-metrics-daemon-vs8ck","openshift-network-operator/iptables-alerter-lhxv8","kube-system/konnectivity-agent-kxfln","openshift-cluster-node-tuning-operator/tuned-k4lpw","openshift-image-registry/node-ca-g9tgf","openshift-network-diagnostics/network-check-target-9b7jj","openshift-ovn-kubernetes/ovnkube-node-98qt9"]
Jun 15 05:48:06.246310 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.246288    2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jun 15 05:48:06.246715 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.246698    2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-16.ec2.internal"
Jun 15 05:48:06.247535 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.247513    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:06.247622 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:06.247580    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9ws7" podUID="8c764213-65e7-4296-a8b6-023c581b36c1"
Jun 15 05:48:06.250031 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.250014    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j6vx"
Jun 15 05:48:06.251938 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.251917    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jun 15 05:48:06.252025 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.251942    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ssvtp\""
Jun 15 05:48:06.252025 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.251923    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jun 15 05:48:06.252175 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.252160    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4dv8m"
Jun 15 05:48:06.252270 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.252256    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jun 15 05:48:06.253836 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.253815    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c764213-65e7-4296-a8b6-023c581b36c1-kubelet-config\") pod \"global-pull-secret-syncer-z9ws7\" (UID: \"8c764213-65e7-4296-a8b6-023c581b36c1\") " pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:06.253948 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.253848    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c764213-65e7-4296-a8b6-023c581b36c1-dbus\") pod \"global-pull-secret-syncer-z9ws7\" (UID: \"8c764213-65e7-4296-a8b6-023c581b36c1\") " pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:06.253948 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.253884    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c764213-65e7-4296-a8b6-023c581b36c1-original-pull-secret\") pod \"global-pull-secret-syncer-z9ws7\" (UID: \"8c764213-65e7-4296-a8b6-023c581b36c1\") " pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:06.254103 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.253970    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jun 15 05:48:06.254103 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.254062    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pl6bc\""
Jun 15 05:48:06.254103 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.254066    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jun 15 05:48:06.254306 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.254288    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jwgfl"
Jun 15 05:48:06.256159 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.256144    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jun 15 05:48:06.256159 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.256157    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jun 15 05:48:06.256386 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.256372    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.256967 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.256952    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5b8nt\""
Jun 15 05:48:06.257039 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.257023    2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jun 15 05:48:06.257089 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.257050    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jun 15 05:48:06.257138 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.257088    2572 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-16.ec2.internal"
Jun 15 05:48:06.257138 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.257093    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jun 15 05:48:06.257232 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.257134    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jun 15 05:48:06.258542 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.258524    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:06.258633 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.258541    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-s829x\""
Jun 15 05:48:06.258633 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.258539    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jun 15 05:48:06.258633 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:06.258583    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vs8ck" podUID="57565893-704b-4974-bc5b-c8f9cb932d15"
Jun 15 05:48:06.258938 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.258920    2572 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jun 15 05:48:06.260707 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.260689    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lhxv8"
Jun 15 05:48:06.262429 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.262411    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jun 15 05:48:06.262531 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.262515    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jun 15 05:48:06.262580 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.262530    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jun 15 05:48:06.262761 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.262748    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fq8hs\""
Jun 15 05:48:06.262821 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.262782    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kxfln"
Jun 15 05:48:06.264580 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.264564    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-llbbn\""
Jun 15 05:48:06.264580 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.264571    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jun 15 05:48:06.264764 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.264752    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jun 15 05:48:06.265605 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.265592    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.267319 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.267306    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jun 15 05:48:06.267458 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.267444    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jun 15 05:48:06.267713 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.267701    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-jww2h\""
Jun 15 05:48:06.267816 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.267804    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g9tgf"
Jun 15 05:48:06.268460 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.268445    2572 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jun 15 05:48:06.269626 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.269606    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jun 15 05:48:06.269719 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.269644    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jun 15 05:48:06.269895 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.269868    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jun 15 05:48:06.269895 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.269888    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wmkz5\""
Jun 15 05:48:06.270041 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.270021    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:06.270132 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:06.270078    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b7jj" podUID="243c1d92-89f7-4ebb-838a-0f18bd3216d8"
Jun 15 05:48:06.272204 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.272187    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-16.ec2.internal"]
Jun 15 05:48:06.272289 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.272216    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-16.ec2.internal"]
Jun 15 05:48:06.272289 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.272286    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.274967 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.274531    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jun 15 05:48:06.274967 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.274882    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jun 15 05:48:06.274967 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.274891    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jun 15 05:48:06.275266 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.275153    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lx7cx\""
Jun 15 05:48:06.275345 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.275331    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jun 15 05:48:06.275726 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.275518    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jun 15 05:48:06.275726 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.275614    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jun 15 05:48:06.280851 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.280828    2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-m26t6"
Jun 15 05:48:06.289736 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.289717    2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-m26t6"
Jun 15 05:48:06.334699 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:06.334464    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf55c91d0f7b296b874c42e3338400df2.slice/crio-43e17a5dcfff39deb49858189c7677dd2082e66e2f85b9f7a87142338e32c69c WatchSource:0}: Error finding container 43e17a5dcfff39deb49858189c7677dd2082e66e2f85b9f7a87142338e32c69c: Status 404 returned error can't find the container with id 43e17a5dcfff39deb49858189c7677dd2082e66e2f85b9f7a87142338e32c69c
Jun 15 05:48:06.334963 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:06.334939    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22301900658e933839cf55f1e43739b1.slice/crio-cba24e8a944061f6c72770dd5b3d16ecce3f0e42f5193980bb0e8201172ef6a7 WatchSource:0}: Error finding container cba24e8a944061f6c72770dd5b3d16ecce3f0e42f5193980bb0e8201172ef6a7: Status 404 returned error can't find the container with id cba24e8a944061f6c72770dd5b3d16ecce3f0e42f5193980bb0e8201172ef6a7
Jun 15 05:48:06.340161 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.340133    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 15 05:48:06.348247 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.348229    2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jun 15 05:48:06.350003 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.349969    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-06-14 05:43:05 +0000 UTC" deadline="2027-11-19 04:06:44.462493599 +0000 UTC"
Jun 15 05:48:06.350003 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.350002    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12526h18m38.112493982s"
Jun 15 05:48:06.354296 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.354274    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-cnibin\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.354401 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.354309    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgrtp\" (UniqueName: \"kubernetes.io/projected/b42bd55d-7f2c-4928-afa4-fc574974b8ae-kube-api-access-cgrtp\") pod \"aws-ebs-csi-driver-node-2j6vx\" (UID: \"b42bd55d-7f2c-4928-afa4-fc574974b8ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j6vx"
Jun 15 05:48:06.354401 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.354333    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-node-log\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.354536 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.354393    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.354536 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.354423    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0313d892-f05a-4331-b972-c991f2ffe06a-iptables-alerter-script\") pod \"iptables-alerter-lhxv8\" (UID: \"0313d892-f05a-4331-b972-c991f2ffe06a\") " pod="openshift-network-operator/iptables-alerter-lhxv8"
Jun 15 05:48:06.354536 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.354440    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57565893-704b-4974-bc5b-c8f9cb932d15-metrics-certs\") pod \"network-metrics-daemon-vs8ck\" (UID: \"57565893-704b-4974-bc5b-c8f9cb932d15\") " pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:06.354536 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.354460    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/60d38c85-2ead-4253-915b-fa8462546084-konnectivity-ca\") pod \"konnectivity-agent-kxfln\" (UID: \"60d38c85-2ead-4253-915b-fa8462546084\") " pod="kube-system/konnectivity-agent-kxfln"
Jun 15 05:48:06.354536 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.354491    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b42bd55d-7f2c-4928-afa4-fc574974b8ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2j6vx\" (UID: \"b42bd55d-7f2c-4928-afa4-fc574974b8ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j6vx"
Jun 15 05:48:06.354771 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.354519    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/634277cb-f6c8-4254-b1aa-ccaebfe829a9-cni-binary-copy\") pod \"multus-additional-cni-plugins-jwgfl\" (UID: \"634277cb-f6c8-4254-b1aa-ccaebfe829a9\") " pod="openshift-multus/multus-additional-cni-plugins-jwgfl"
Jun 15 05:48:06.354771 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.354577    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scpx8\" (UniqueName: \"kubernetes.io/projected/634277cb-f6c8-4254-b1aa-ccaebfe829a9-kube-api-access-scpx8\") pod \"multus-additional-cni-plugins-jwgfl\" (UID: \"634277cb-f6c8-4254-b1aa-ccaebfe829a9\") " pod="openshift-multus/multus-additional-cni-plugins-jwgfl"
Jun 15 05:48:06.354771 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.354601    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mc2s\" (UniqueName: \"kubernetes.io/projected/0313d892-f05a-4331-b972-c991f2ffe06a-kube-api-access-2mc2s\") pod \"iptables-alerter-lhxv8\" (UID: \"0313d892-f05a-4331-b972-c991f2ffe06a\") " pod="openshift-network-operator/iptables-alerter-lhxv8"
Jun 15 05:48:06.354771 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.354630    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c764213-65e7-4296-a8b6-023c581b36c1-dbus\") pod \"global-pull-secret-syncer-z9ws7\" (UID: \"8c764213-65e7-4296-a8b6-023c581b36c1\") " pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:06.354771 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.354654    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-os-release\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.354771 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.354678    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-host-var-lib-kubelet\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.354771 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.354698    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-hostroot\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.354771 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.354724    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8c764213-65e7-4296-a8b6-023c581b36c1-dbus\") pod \"global-pull-secret-syncer-z9ws7\" (UID: \"8c764213-65e7-4296-a8b6-023c581b36c1\") " pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:06.354771 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.354732    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg9d5\" (UniqueName: \"kubernetes.io/projected/d313c84b-5ccf-445d-bae0-2bc39eea2db2-kube-api-access-jg9d5\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.354771 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.354775    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-etc-kubernetes\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.355195 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.354803    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48e95765-7c73-4f62-a381-bb60c8a04b1b-tmp\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.355195 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.354831    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-systemd-units\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.355195 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.354850    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-host-slash\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.355195 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.354865    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-host-cni-netd\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.355195 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.354899    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-multus-socket-dir-parent\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.355195 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.354928    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-host-var-lib-cni-multus\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.355195 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.354967    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d313c84b-5ccf-445d-bae0-2bc39eea2db2-multus-daemon-config\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.355195 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355012    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-etc-sysctl-conf\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.355195 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355043    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b5eacff-8b85-4b48-a883-22d29645244a-ovnkube-config\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.355195 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355066    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mws52\" (UniqueName: \"kubernetes.io/projected/243c1d92-89f7-4ebb-838a-0f18bd3216d8-kube-api-access-mws52\") pod \"network-check-target-9b7jj\" (UID: \"243c1d92-89f7-4ebb-838a-0f18bd3216d8\") " pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:06.355195 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355092    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b42bd55d-7f2c-4928-afa4-fc574974b8ae-socket-dir\") pod \"aws-ebs-csi-driver-node-2j6vx\" (UID: \"b42bd55d-7f2c-4928-afa4-fc574974b8ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j6vx"
Jun 15 05:48:06.355195 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355115    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/634277cb-f6c8-4254-b1aa-ccaebfe829a9-os-release\") pod \"multus-additional-cni-plugins-jwgfl\" (UID: \"634277cb-f6c8-4254-b1aa-ccaebfe829a9\") " pod="openshift-multus/multus-additional-cni-plugins-jwgfl"
Jun 15 05:48:06.355195 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355139    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/634277cb-f6c8-4254-b1aa-ccaebfe829a9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jwgfl\" (UID: \"634277cb-f6c8-4254-b1aa-ccaebfe829a9\") " pod="openshift-multus/multus-additional-cni-plugins-jwgfl"
Jun 15 05:48:06.355195 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355182    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b26hg\" (UniqueName: \"kubernetes.io/projected/c80610bc-c188-476f-8a0c-ce642967f28e-kube-api-access-b26hg\") pod \"node-ca-g9tgf\" (UID: \"c80610bc-c188-476f-8a0c-ce642967f28e\") " pod="openshift-image-registry/node-ca-g9tgf"
Jun 15 05:48:06.355195 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355206    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-multus-cni-dir\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.355930 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355229    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/60d38c85-2ead-4253-915b-fa8462546084-agent-certs\") pod \"konnectivity-agent-kxfln\" (UID: \"60d38c85-2ead-4253-915b-fa8462546084\") " pod="kube-system/konnectivity-agent-kxfln"
Jun 15 05:48:06.355930 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355250    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/634277cb-f6c8-4254-b1aa-ccaebfe829a9-cnibin\") pod \"multus-additional-cni-plugins-jwgfl\" (UID: \"634277cb-f6c8-4254-b1aa-ccaebfe829a9\") " pod="openshift-multus/multus-additional-cni-plugins-jwgfl"
Jun 15 05:48:06.355930 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355272    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-etc-modprobe-d\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.355930 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355296    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-system-cni-dir\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.355930 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355321    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-run\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.355930 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355343    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-var-lib-kubelet\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.355930 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355400    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c80610bc-c188-476f-8a0c-ce642967f28e-host\") pod \"node-ca-g9tgf\" (UID: \"c80610bc-c188-476f-8a0c-ce642967f28e\") " pod="openshift-image-registry/node-ca-g9tgf"
Jun 15 05:48:06.355930 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355424    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-host-run-ovn-kubernetes\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.355930 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355448    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5l59\" (UniqueName: \"kubernetes.io/projected/54e26677-5db1-4e31-94c1-a5d3c5db5047-kube-api-access-x5l59\") pod \"node-resolver-4dv8m\" (UID: \"54e26677-5db1-4e31-94c1-a5d3c5db5047\") " pod="openshift-dns/node-resolver-4dv8m"
Jun 15 05:48:06.355930 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355471    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-host-var-lib-cni-bin\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.355930 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355509    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0313d892-f05a-4331-b972-c991f2ffe06a-host-slash\") pod \"iptables-alerter-lhxv8\" (UID: \"0313d892-f05a-4331-b972-c991f2ffe06a\") " pod="openshift-network-operator/iptables-alerter-lhxv8"
Jun 15 05:48:06.355930 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355541    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-etc-systemd\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.355930 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355567    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-sys\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.355930 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355589    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zbpv\" (UniqueName: \"kubernetes.io/projected/48e95765-7c73-4f62-a381-bb60c8a04b1b-kube-api-access-6zbpv\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.355930 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355611    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d313c84b-5ccf-445d-bae0-2bc39eea2db2-cni-binary-copy\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.355930 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355634    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b5eacff-8b85-4b48-a883-22d29645244a-ovn-node-metrics-cert\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.355930 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355656    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nksk\" (UniqueName: \"kubernetes.io/projected/0b5eacff-8b85-4b48-a883-22d29645244a-kube-api-access-9nksk\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.356651 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355678    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-run-systemd\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.356651 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355705    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-var-lib-openvswitch\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.356651 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355731    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-run-openvswitch\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.356651 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355746    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b5eacff-8b85-4b48-a883-22d29645244a-env-overrides\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.356651 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355762    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c80610bc-c188-476f-8a0c-ce642967f28e-serviceca\") pod \"node-ca-g9tgf\" (UID: \"c80610bc-c188-476f-8a0c-ce642967f28e\") " pod="openshift-image-registry/node-ca-g9tgf"
Jun 15 05:48:06.356651 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355779    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c764213-65e7-4296-a8b6-023c581b36c1-kubelet-config\") pod \"global-pull-secret-syncer-z9ws7\" (UID: \"8c764213-65e7-4296-a8b6-023c581b36c1\") " pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:06.356651 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355793    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b5eacff-8b85-4b48-a883-22d29645244a-ovnkube-script-lib\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.356651 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355807    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/634277cb-f6c8-4254-b1aa-ccaebfe829a9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jwgfl\" (UID: \"634277cb-f6c8-4254-b1aa-ccaebfe829a9\") " pod="openshift-multus/multus-additional-cni-plugins-jwgfl"
Jun 15 05:48:06.356651 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355820    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-host\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.356651 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355835    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b42bd55d-7f2c-4928-afa4-fc574974b8ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-2j6vx\" (UID: \"b42bd55d-7f2c-4928-afa4-fc574974b8ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j6vx"
Jun 15 05:48:06.356651 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355865    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c764213-65e7-4296-a8b6-023c581b36c1-original-pull-secret\") pod \"global-pull-secret-syncer-z9ws7\" (UID: \"8c764213-65e7-4296-a8b6-023c581b36c1\") " pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:06.356651 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355886    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-host-run-netns\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.356651 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355888    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8c764213-65e7-4296-a8b6-023c581b36c1-kubelet-config\") pod \"global-pull-secret-syncer-z9ws7\" (UID: \"8c764213-65e7-4296-a8b6-023c581b36c1\") " pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:06.356651 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355900    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/54e26677-5db1-4e31-94c1-a5d3c5db5047-hosts-file\") pod \"node-resolver-4dv8m\" (UID: \"54e26677-5db1-4e31-94c1-a5d3c5db5047\") " pod="openshift-dns/node-resolver-4dv8m"
Jun 15 05:48:06.356651 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355913    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-lib-modules\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.356651 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355933    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-host-kubelet\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.356651 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.355968    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54e26677-5db1-4e31-94c1-a5d3c5db5047-tmp-dir\") pod \"node-resolver-4dv8m\" (UID: \"54e26677-5db1-4e31-94c1-a5d3c5db5047\") " pod="openshift-dns/node-resolver-4dv8m"
Jun 15 05:48:06.357125 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.356030    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-host-run-netns\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.357125 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.356056    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-multus-conf-dir\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.357125 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:06.356075    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 15 05:48:06.357125 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.356079    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d69m8\" (UniqueName: \"kubernetes.io/projected/57565893-704b-4974-bc5b-c8f9cb932d15-kube-api-access-d69m8\") pod \"network-metrics-daemon-vs8ck\" (UID: \"57565893-704b-4974-bc5b-c8f9cb932d15\") " pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:06.357125 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.356102    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b42bd55d-7f2c-4928-afa4-fc574974b8ae-sys-fs\") pod \"aws-ebs-csi-driver-node-2j6vx\" (UID: \"b42bd55d-7f2c-4928-afa4-fc574974b8ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j6vx"
Jun 15 05:48:06.357125 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:06.356135    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c764213-65e7-4296-a8b6-023c581b36c1-original-pull-secret podName:8c764213-65e7-4296-a8b6-023c581b36c1 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:06.856113428 +0000 UTC m=+2.004293411 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c764213-65e7-4296-a8b6-023c581b36c1-original-pull-secret") pod "global-pull-secret-syncer-z9ws7" (UID: "8c764213-65e7-4296-a8b6-023c581b36c1") : object "kube-system"/"original-pull-secret" not registered
Jun 15 05:48:06.357125 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.356156    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/634277cb-f6c8-4254-b1aa-ccaebfe829a9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jwgfl\" (UID: \"634277cb-f6c8-4254-b1aa-ccaebfe829a9\") " pod="openshift-multus/multus-additional-cni-plugins-jwgfl"
Jun 15 05:48:06.357125 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.356202    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-log-socket\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.357125 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.356229    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-etc-kubernetes\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.357125 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.356254    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/634277cb-f6c8-4254-b1aa-ccaebfe829a9-system-cni-dir\") pod \"multus-additional-cni-plugins-jwgfl\" (UID: \"634277cb-f6c8-4254-b1aa-ccaebfe829a9\") " pod="openshift-multus/multus-additional-cni-plugins-jwgfl"
Jun 15 05:48:06.357125 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.356279    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-etc-sysctl-d\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.357125 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.356305    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/48e95765-7c73-4f62-a381-bb60c8a04b1b-etc-tuned\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.357125 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.356327    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-run-ovn\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.357125 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.356383    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-host-cni-bin\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.357125 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.356411    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b42bd55d-7f2c-4928-afa4-fc574974b8ae-registration-dir\") pod \"aws-ebs-csi-driver-node-2j6vx\" (UID: \"b42bd55d-7f2c-4928-afa4-fc574974b8ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j6vx"
Jun 15 05:48:06.357125 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.356435    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b42bd55d-7f2c-4928-afa4-fc574974b8ae-device-dir\") pod \"aws-ebs-csi-driver-node-2j6vx\" (UID: \"b42bd55d-7f2c-4928-afa4-fc574974b8ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j6vx"
Jun 15 05:48:06.357583 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.356461    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-etc-openvswitch\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.357583 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.356483    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-host-run-k8s-cni-cncf-io\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.357583 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.356506    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-host-run-multus-certs\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.357583 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.356529    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-etc-sysconfig\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.457114 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457022    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nksk\" (UniqueName: \"kubernetes.io/projected/0b5eacff-8b85-4b48-a883-22d29645244a-kube-api-access-9nksk\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.457304 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457131    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-run-systemd\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.457304 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457160    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-var-lib-openvswitch\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.457304 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457183    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-run-openvswitch\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.457304 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457230    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-run-openvswitch\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.457304 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457258    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-var-lib-openvswitch\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.457304 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457271    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-run-systemd\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.457304 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457290    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b5eacff-8b85-4b48-a883-22d29645244a-env-overrides\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.457304 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457305    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c80610bc-c188-476f-8a0c-ce642967f28e-serviceca\") pod \"node-ca-g9tgf\" (UID: \"c80610bc-c188-476f-8a0c-ce642967f28e\") " pod="openshift-image-registry/node-ca-g9tgf"
Jun 15 05:48:06.457709 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457321    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b5eacff-8b85-4b48-a883-22d29645244a-ovnkube-script-lib\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.457709 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457335    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/634277cb-f6c8-4254-b1aa-ccaebfe829a9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jwgfl\" (UID: \"634277cb-f6c8-4254-b1aa-ccaebfe829a9\") " pod="openshift-multus/multus-additional-cni-plugins-jwgfl"
Jun 15 05:48:06.457709 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457373    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-host\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.457709 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457396    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b42bd55d-7f2c-4928-afa4-fc574974b8ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-2j6vx\" (UID: \"b42bd55d-7f2c-4928-afa4-fc574974b8ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j6vx"
Jun 15 05:48:06.457709 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457430    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-host-run-netns\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.457709 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457452    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/54e26677-5db1-4e31-94c1-a5d3c5db5047-hosts-file\") pod \"node-resolver-4dv8m\" (UID: \"54e26677-5db1-4e31-94c1-a5d3c5db5047\") " pod="openshift-dns/node-resolver-4dv8m"
Jun 15 05:48:06.457709 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457472    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-lib-modules\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.457709 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457480    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-host-run-netns\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.457709 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457491    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-host-kubelet\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.457709 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457500    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b42bd55d-7f2c-4928-afa4-fc574974b8ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-2j6vx\" (UID: \"b42bd55d-7f2c-4928-afa4-fc574974b8ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j6vx"
Jun 15 05:48:06.457709 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457514    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54e26677-5db1-4e31-94c1-a5d3c5db5047-tmp-dir\") pod \"node-resolver-4dv8m\" (UID: \"54e26677-5db1-4e31-94c1-a5d3c5db5047\") " pod="openshift-dns/node-resolver-4dv8m"
Jun 15 05:48:06.457709 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457536    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-host-run-netns\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.457709 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457541    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/54e26677-5db1-4e31-94c1-a5d3c5db5047-hosts-file\") pod \"node-resolver-4dv8m\" (UID: \"54e26677-5db1-4e31-94c1-a5d3c5db5047\") " pod="openshift-dns/node-resolver-4dv8m"
Jun 15 05:48:06.457709 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457559    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-multus-conf-dir\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.457709 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457565    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/634277cb-f6c8-4254-b1aa-ccaebfe829a9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jwgfl\" (UID: \"634277cb-f6c8-4254-b1aa-ccaebfe829a9\") " pod="openshift-multus/multus-additional-cni-plugins-jwgfl"
Jun 15 05:48:06.457709 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457582    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d69m8\" (UniqueName: \"kubernetes.io/projected/57565893-704b-4974-bc5b-c8f9cb932d15-kube-api-access-d69m8\") pod \"network-metrics-daemon-vs8ck\" (UID: \"57565893-704b-4974-bc5b-c8f9cb932d15\") " pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:06.457709 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457616    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-host-run-netns\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.457709 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457631    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-lib-modules\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.458601 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457640    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b42bd55d-7f2c-4928-afa4-fc574974b8ae-sys-fs\") pod \"aws-ebs-csi-driver-node-2j6vx\" (UID: \"b42bd55d-7f2c-4928-afa4-fc574974b8ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j6vx"
Jun 15 05:48:06.458601 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457661    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-multus-conf-dir\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.458601 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457669    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/634277cb-f6c8-4254-b1aa-ccaebfe829a9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jwgfl\" (UID: \"634277cb-f6c8-4254-b1aa-ccaebfe829a9\") " pod="openshift-multus/multus-additional-cni-plugins-jwgfl"
Jun 15 05:48:06.458601 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457694    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-log-socket\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.458601 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457713    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b42bd55d-7f2c-4928-afa4-fc574974b8ae-sys-fs\") pod \"aws-ebs-csi-driver-node-2j6vx\" (UID: \"b42bd55d-7f2c-4928-afa4-fc574974b8ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j6vx"
Jun 15 05:48:06.458601 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457719    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-etc-kubernetes\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.458601 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457745    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/634277cb-f6c8-4254-b1aa-ccaebfe829a9-system-cni-dir\") pod \"multus-additional-cni-plugins-jwgfl\" (UID: \"634277cb-f6c8-4254-b1aa-ccaebfe829a9\") " pod="openshift-multus/multus-additional-cni-plugins-jwgfl"
Jun 15 05:48:06.458601 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457771    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-etc-sysctl-d\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.458601 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457795    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/48e95765-7c73-4f62-a381-bb60c8a04b1b-etc-tuned\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.458601 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457817    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-run-ovn\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.458601 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457845    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-host-cni-bin\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.458601 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457856    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54e26677-5db1-4e31-94c1-a5d3c5db5047-tmp-dir\") pod \"node-resolver-4dv8m\" (UID: \"54e26677-5db1-4e31-94c1-a5d3c5db5047\") " pod="openshift-dns/node-resolver-4dv8m"
Jun 15 05:48:06.458601 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457875    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b42bd55d-7f2c-4928-afa4-fc574974b8ae-registration-dir\") pod \"aws-ebs-csi-driver-node-2j6vx\" (UID: \"b42bd55d-7f2c-4928-afa4-fc574974b8ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j6vx"
Jun 15 05:48:06.458601 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457886    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c80610bc-c188-476f-8a0c-ce642967f28e-serviceca\") pod \"node-ca-g9tgf\" (UID: \"c80610bc-c188-476f-8a0c-ce642967f28e\") " pod="openshift-image-registry/node-ca-g9tgf"
Jun 15 05:48:06.458601 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457895    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/634277cb-f6c8-4254-b1aa-ccaebfe829a9-system-cni-dir\") pod \"multus-additional-cni-plugins-jwgfl\" (UID: \"634277cb-f6c8-4254-b1aa-ccaebfe829a9\") " pod="openshift-multus/multus-additional-cni-plugins-jwgfl"
Jun 15 05:48:06.458601 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457901    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b42bd55d-7f2c-4928-afa4-fc574974b8ae-device-dir\") pod \"aws-ebs-csi-driver-node-2j6vx\" (UID: \"b42bd55d-7f2c-4928-afa4-fc574974b8ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j6vx"
Jun 15 05:48:06.458601 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457944    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-etc-openvswitch\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.459452 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457946    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b42bd55d-7f2c-4928-afa4-fc574974b8ae-device-dir\") pod \"aws-ebs-csi-driver-node-2j6vx\" (UID: \"b42bd55d-7f2c-4928-afa4-fc574974b8ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j6vx"
Jun 15 05:48:06.459452 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457973    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0b5eacff-8b85-4b48-a883-22d29645244a-env-overrides\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.459452 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457987    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-run-ovn\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.459452 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457999    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-log-socket\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.459452 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.457998    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b42bd55d-7f2c-4928-afa4-fc574974b8ae-registration-dir\") pod \"aws-ebs-csi-driver-node-2j6vx\" (UID: \"b42bd55d-7f2c-4928-afa4-fc574974b8ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j6vx"
Jun 15 05:48:06.459452 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458026    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-etc-openvswitch\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.459452 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458039    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-host\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.459452 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458047    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0b5eacff-8b85-4b48-a883-22d29645244a-ovnkube-script-lib\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.459452 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458058    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-host-run-k8s-cni-cncf-io\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.459452 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458108    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-host-run-k8s-cni-cncf-io\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.459452 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458106    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-host-run-multus-certs\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.459452 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458055    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-host-cni-bin\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.459452 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458143    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-etc-sysconfig\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.459452 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458148    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-host-run-multus-certs\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.459452 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458124    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-etc-kubernetes\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.459452 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458168    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-cnibin\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.459452 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458186    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-etc-sysctl-d\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.459452 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458199    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-etc-sysconfig\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.460352 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458204    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgrtp\" (UniqueName: \"kubernetes.io/projected/b42bd55d-7f2c-4928-afa4-fc574974b8ae-kube-api-access-cgrtp\") pod \"aws-ebs-csi-driver-node-2j6vx\" (UID: \"b42bd55d-7f2c-4928-afa4-fc574974b8ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j6vx"
Jun 15 05:48:06.460352 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458219    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/634277cb-f6c8-4254-b1aa-ccaebfe829a9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jwgfl\" (UID: \"634277cb-f6c8-4254-b1aa-ccaebfe829a9\") " pod="openshift-multus/multus-additional-cni-plugins-jwgfl"
Jun 15 05:48:06.460352 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458231    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-node-log\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.460352 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458209    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-cnibin\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.460352 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458252    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-host-kubelet\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.460352 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458257    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.460352 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458277    2572 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jun 15 05:48:06.460352 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458290    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-node-log\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.460352 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458297    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0313d892-f05a-4331-b972-c991f2ffe06a-iptables-alerter-script\") pod \"iptables-alerter-lhxv8\" (UID: \"0313d892-f05a-4331-b972-c991f2ffe06a\") " pod="openshift-network-operator/iptables-alerter-lhxv8"
Jun 15 05:48:06.460352 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458320    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57565893-704b-4974-bc5b-c8f9cb932d15-metrics-certs\") pod \"network-metrics-daemon-vs8ck\" (UID: \"57565893-704b-4974-bc5b-c8f9cb932d15\") " pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:06.460352 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458327    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.460352 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:06.458410    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:48:06.460352 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:06.458467    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57565893-704b-4974-bc5b-c8f9cb932d15-metrics-certs podName:57565893-704b-4974-bc5b-c8f9cb932d15 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:06.958451155 +0000 UTC m=+2.106631145 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57565893-704b-4974-bc5b-c8f9cb932d15-metrics-certs") pod "network-metrics-daemon-vs8ck" (UID: "57565893-704b-4974-bc5b-c8f9cb932d15") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:48:06.460352 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458410    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/60d38c85-2ead-4253-915b-fa8462546084-konnectivity-ca\") pod \"konnectivity-agent-kxfln\" (UID: \"60d38c85-2ead-4253-915b-fa8462546084\") " pod="kube-system/konnectivity-agent-kxfln"
Jun 15 05:48:06.460352 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458514    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b42bd55d-7f2c-4928-afa4-fc574974b8ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2j6vx\" (UID: \"b42bd55d-7f2c-4928-afa4-fc574974b8ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j6vx"
Jun 15 05:48:06.460352 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458544    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/634277cb-f6c8-4254-b1aa-ccaebfe829a9-cni-binary-copy\") pod \"multus-additional-cni-plugins-jwgfl\" (UID: \"634277cb-f6c8-4254-b1aa-ccaebfe829a9\") " pod="openshift-multus/multus-additional-cni-plugins-jwgfl"
Jun 15 05:48:06.460352 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458571    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scpx8\" (UniqueName: \"kubernetes.io/projected/634277cb-f6c8-4254-b1aa-ccaebfe829a9-kube-api-access-scpx8\") pod \"multus-additional-cni-plugins-jwgfl\" (UID: \"634277cb-f6c8-4254-b1aa-ccaebfe829a9\") " pod="openshift-multus/multus-additional-cni-plugins-jwgfl"
Jun 15 05:48:06.461098 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458595    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mc2s\" (UniqueName: \"kubernetes.io/projected/0313d892-f05a-4331-b972-c991f2ffe06a-kube-api-access-2mc2s\") pod \"iptables-alerter-lhxv8\" (UID: \"0313d892-f05a-4331-b972-c991f2ffe06a\") " pod="openshift-network-operator/iptables-alerter-lhxv8"
Jun 15 05:48:06.461098 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458605    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b42bd55d-7f2c-4928-afa4-fc574974b8ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2j6vx\" (UID: \"b42bd55d-7f2c-4928-afa4-fc574974b8ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j6vx"
Jun 15 05:48:06.461098 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458618    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-os-release\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.461098 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458640    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-host-var-lib-kubelet\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.461098 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458708    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-os-release\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.461098 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458786    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-hostroot\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.461098 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458819    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg9d5\" (UniqueName: \"kubernetes.io/projected/d313c84b-5ccf-445d-bae0-2bc39eea2db2-kube-api-access-jg9d5\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.461098 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458850    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0313d892-f05a-4331-b972-c991f2ffe06a-iptables-alerter-script\") pod \"iptables-alerter-lhxv8\" (UID: \"0313d892-f05a-4331-b972-c991f2ffe06a\") " pod="openshift-network-operator/iptables-alerter-lhxv8"
Jun 15 05:48:06.461098 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458853    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-etc-kubernetes\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.461098 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458883    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-hostroot\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.461098 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458895    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48e95765-7c73-4f62-a381-bb60c8a04b1b-tmp\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.461098 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458899    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/60d38c85-2ead-4253-915b-fa8462546084-konnectivity-ca\") pod \"konnectivity-agent-kxfln\" (UID: \"60d38c85-2ead-4253-915b-fa8462546084\") " pod="kube-system/konnectivity-agent-kxfln"
Jun 15 05:48:06.461098 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458907    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-host-var-lib-kubelet\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.461098 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458919    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-systemd-units\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.461098 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458942    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-host-slash\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.461098 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458964    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-systemd-units\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.461098 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458902    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-etc-kubernetes\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.461098 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.458983    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-host-cni-netd\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.462047 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459011    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-multus-socket-dir-parent\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.462047 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459022    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-host-slash\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.462047 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459039    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-host-var-lib-cni-multus\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.462047 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459059    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-host-cni-netd\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.462047 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459063    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d313c84b-5ccf-445d-bae0-2bc39eea2db2-multus-daemon-config\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.462047 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459111    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-multus-socket-dir-parent\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.462047 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459119    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-host-var-lib-cni-multus\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.462047 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459143    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-etc-sysctl-conf\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.462047 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459168    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b5eacff-8b85-4b48-a883-22d29645244a-ovnkube-config\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.462047 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459181    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/634277cb-f6c8-4254-b1aa-ccaebfe829a9-cni-binary-copy\") pod \"multus-additional-cni-plugins-jwgfl\" (UID: \"634277cb-f6c8-4254-b1aa-ccaebfe829a9\") " pod="openshift-multus/multus-additional-cni-plugins-jwgfl"
Jun 15 05:48:06.462047 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459191    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mws52\" (UniqueName: \"kubernetes.io/projected/243c1d92-89f7-4ebb-838a-0f18bd3216d8-kube-api-access-mws52\") pod \"network-check-target-9b7jj\" (UID: \"243c1d92-89f7-4ebb-838a-0f18bd3216d8\") " pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:06.462047 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459214    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b42bd55d-7f2c-4928-afa4-fc574974b8ae-socket-dir\") pod \"aws-ebs-csi-driver-node-2j6vx\" (UID: \"b42bd55d-7f2c-4928-afa4-fc574974b8ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j6vx"
Jun 15 05:48:06.462047 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459238    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/634277cb-f6c8-4254-b1aa-ccaebfe829a9-os-release\") pod \"multus-additional-cni-plugins-jwgfl\" (UID: \"634277cb-f6c8-4254-b1aa-ccaebfe829a9\") " pod="openshift-multus/multus-additional-cni-plugins-jwgfl"
Jun 15 05:48:06.462047 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459259    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/634277cb-f6c8-4254-b1aa-ccaebfe829a9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jwgfl\" (UID: \"634277cb-f6c8-4254-b1aa-ccaebfe829a9\") " pod="openshift-multus/multus-additional-cni-plugins-jwgfl"
Jun 15 05:48:06.462047 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459280    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b26hg\" (UniqueName: \"kubernetes.io/projected/c80610bc-c188-476f-8a0c-ce642967f28e-kube-api-access-b26hg\") pod \"node-ca-g9tgf\" (UID: \"c80610bc-c188-476f-8a0c-ce642967f28e\") " pod="openshift-image-registry/node-ca-g9tgf"
Jun 15 05:48:06.462047 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459304    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-multus-cni-dir\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.462047 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459321    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-etc-sysctl-conf\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.462667 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459328    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/60d38c85-2ead-4253-915b-fa8462546084-agent-certs\") pod \"konnectivity-agent-kxfln\" (UID: \"60d38c85-2ead-4253-915b-fa8462546084\") " pod="kube-system/konnectivity-agent-kxfln"
Jun 15 05:48:06.462667 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459353    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/634277cb-f6c8-4254-b1aa-ccaebfe829a9-cnibin\") pod \"multus-additional-cni-plugins-jwgfl\" (UID: \"634277cb-f6c8-4254-b1aa-ccaebfe829a9\") " pod="openshift-multus/multus-additional-cni-plugins-jwgfl"
Jun 15 05:48:06.462667 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459394    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-etc-modprobe-d\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.462667 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459396    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/634277cb-f6c8-4254-b1aa-ccaebfe829a9-os-release\") pod \"multus-additional-cni-plugins-jwgfl\" (UID: \"634277cb-f6c8-4254-b1aa-ccaebfe829a9\") " pod="openshift-multus/multus-additional-cni-plugins-jwgfl"
Jun 15 05:48:06.462667 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459416    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-system-cni-dir\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.462667 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459440    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-run\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.462667 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459462    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-var-lib-kubelet\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.462667 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459483    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c80610bc-c188-476f-8a0c-ce642967f28e-host\") pod \"node-ca-g9tgf\" (UID: \"c80610bc-c188-476f-8a0c-ce642967f28e\") " pod="openshift-image-registry/node-ca-g9tgf"
Jun 15 05:48:06.462667 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459506    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-host-run-ovn-kubernetes\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.462667 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459532    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5l59\" (UniqueName: \"kubernetes.io/projected/54e26677-5db1-4e31-94c1-a5d3c5db5047-kube-api-access-x5l59\") pod \"node-resolver-4dv8m\" (UID: \"54e26677-5db1-4e31-94c1-a5d3c5db5047\") " pod="openshift-dns/node-resolver-4dv8m"
Jun 15 05:48:06.462667 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459556    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-host-var-lib-cni-bin\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.462667 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459579    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0313d892-f05a-4331-b972-c991f2ffe06a-host-slash\") pod \"iptables-alerter-lhxv8\" (UID: \"0313d892-f05a-4331-b972-c991f2ffe06a\") " pod="openshift-network-operator/iptables-alerter-lhxv8"
Jun 15 05:48:06.462667 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459601    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-etc-systemd\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.462667 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459624    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-sys\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.462667 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459644    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/d313c84b-5ccf-445d-bae0-2bc39eea2db2-multus-daemon-config\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.462667 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459652    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b42bd55d-7f2c-4928-afa4-fc574974b8ae-socket-dir\") pod \"aws-ebs-csi-driver-node-2j6vx\" (UID: \"b42bd55d-7f2c-4928-afa4-fc574974b8ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j6vx"
Jun 15 05:48:06.462667 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459647    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zbpv\" (UniqueName: \"kubernetes.io/projected/48e95765-7c73-4f62-a381-bb60c8a04b1b-kube-api-access-6zbpv\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.462667 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459694    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d313c84b-5ccf-445d-bae0-2bc39eea2db2-cni-binary-copy\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.463199 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459707    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0b5eacff-8b85-4b48-a883-22d29645244a-ovnkube-config\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.463199 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459721    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b5eacff-8b85-4b48-a883-22d29645244a-host-run-ovn-kubernetes\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.463199 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459727    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b5eacff-8b85-4b48-a883-22d29645244a-ovn-node-metrics-cert\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.463199 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459902    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-host-var-lib-cni-bin\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.463199 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459949    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0313d892-f05a-4331-b972-c991f2ffe06a-host-slash\") pod \"iptables-alerter-lhxv8\" (UID: \"0313d892-f05a-4331-b972-c991f2ffe06a\") " pod="openshift-network-operator/iptables-alerter-lhxv8"
Jun 15 05:48:06.463199 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.459999    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-etc-systemd\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.463199 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.460054    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-sys\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.463199 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.460070    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-multus-cni-dir\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.463199 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.460099    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/634277cb-f6c8-4254-b1aa-ccaebfe829a9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jwgfl\" (UID: \"634277cb-f6c8-4254-b1aa-ccaebfe829a9\") " pod="openshift-multus/multus-additional-cni-plugins-jwgfl"
Jun 15 05:48:06.463199 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.460124    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/634277cb-f6c8-4254-b1aa-ccaebfe829a9-cnibin\") pod \"multus-additional-cni-plugins-jwgfl\" (UID: \"634277cb-f6c8-4254-b1aa-ccaebfe829a9\") " pod="openshift-multus/multus-additional-cni-plugins-jwgfl"
Jun 15 05:48:06.463199 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.460146    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-etc-modprobe-d\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.463199 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.460152    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-run\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.463199 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.460178    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d313c84b-5ccf-445d-bae0-2bc39eea2db2-system-cni-dir\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.463199 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.460183    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/48e95765-7c73-4f62-a381-bb60c8a04b1b-var-lib-kubelet\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.463199 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.460222    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c80610bc-c188-476f-8a0c-ce642967f28e-host\") pod \"node-ca-g9tgf\" (UID: \"c80610bc-c188-476f-8a0c-ce642967f28e\") " pod="openshift-image-registry/node-ca-g9tgf"
Jun 15 05:48:06.463199 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.460269    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d313c84b-5ccf-445d-bae0-2bc39eea2db2-cni-binary-copy\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.463199 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.461286    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48e95765-7c73-4f62-a381-bb60c8a04b1b-tmp\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.463199 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.461318    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/48e95765-7c73-4f62-a381-bb60c8a04b1b-etc-tuned\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.463724 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.462586    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/60d38c85-2ead-4253-915b-fa8462546084-agent-certs\") pod \"konnectivity-agent-kxfln\" (UID: \"60d38c85-2ead-4253-915b-fa8462546084\") " pod="kube-system/konnectivity-agent-kxfln"
Jun 15 05:48:06.463724 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.462591    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0b5eacff-8b85-4b48-a883-22d29645244a-ovn-node-metrics-cert\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.472962 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.472923    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nksk\" (UniqueName: \"kubernetes.io/projected/0b5eacff-8b85-4b48-a883-22d29645244a-kube-api-access-9nksk\") pod \"ovnkube-node-98qt9\" (UID: \"0b5eacff-8b85-4b48-a883-22d29645244a\") " pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.473781 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:06.473547    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 15 05:48:06.473781 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:06.473569    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 15 05:48:06.473781 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:06.473583    2572 projected.go:196] Error preparing data for projected volume kube-api-access-mws52 for pod openshift-network-diagnostics/network-check-target-9b7jj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:48:06.473781 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:06.473654    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/243c1d92-89f7-4ebb-838a-0f18bd3216d8-kube-api-access-mws52 podName:243c1d92-89f7-4ebb-838a-0f18bd3216d8 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:06.973637246 +0000 UTC m=+2.121817223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mws52" (UniqueName: "kubernetes.io/projected/243c1d92-89f7-4ebb-838a-0f18bd3216d8-kube-api-access-mws52") pod "network-check-target-9b7jj" (UID: "243c1d92-89f7-4ebb-838a-0f18bd3216d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:48:06.475018 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.474993    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5l59\" (UniqueName: \"kubernetes.io/projected/54e26677-5db1-4e31-94c1-a5d3c5db5047-kube-api-access-x5l59\") pod \"node-resolver-4dv8m\" (UID: \"54e26677-5db1-4e31-94c1-a5d3c5db5047\") " pod="openshift-dns/node-resolver-4dv8m"
Jun 15 05:48:06.475425 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.475399    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zbpv\" (UniqueName: \"kubernetes.io/projected/48e95765-7c73-4f62-a381-bb60c8a04b1b-kube-api-access-6zbpv\") pod \"tuned-k4lpw\" (UID: \"48e95765-7c73-4f62-a381-bb60c8a04b1b\") " pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.475540 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.475453    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgrtp\" (UniqueName: \"kubernetes.io/projected/b42bd55d-7f2c-4928-afa4-fc574974b8ae-kube-api-access-cgrtp\") pod \"aws-ebs-csi-driver-node-2j6vx\" (UID: \"b42bd55d-7f2c-4928-afa4-fc574974b8ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j6vx"
Jun 15 05:48:06.475714 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.475696    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg9d5\" (UniqueName: \"kubernetes.io/projected/d313c84b-5ccf-445d-bae0-2bc39eea2db2-kube-api-access-jg9d5\") pod \"multus-ccxxl\" (UID: \"d313c84b-5ccf-445d-bae0-2bc39eea2db2\") " pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.476502 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.476482    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b26hg\" (UniqueName: \"kubernetes.io/projected/c80610bc-c188-476f-8a0c-ce642967f28e-kube-api-access-b26hg\") pod \"node-ca-g9tgf\" (UID: \"c80610bc-c188-476f-8a0c-ce642967f28e\") " pod="openshift-image-registry/node-ca-g9tgf"
Jun 15 05:48:06.476600 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.476577    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d69m8\" (UniqueName: \"kubernetes.io/projected/57565893-704b-4974-bc5b-c8f9cb932d15-kube-api-access-d69m8\") pod \"network-metrics-daemon-vs8ck\" (UID: \"57565893-704b-4974-bc5b-c8f9cb932d15\") " pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:06.477187 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.477169    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mc2s\" (UniqueName: \"kubernetes.io/projected/0313d892-f05a-4331-b972-c991f2ffe06a-kube-api-access-2mc2s\") pod \"iptables-alerter-lhxv8\" (UID: \"0313d892-f05a-4331-b972-c991f2ffe06a\") " pod="openshift-network-operator/iptables-alerter-lhxv8"
Jun 15 05:48:06.477243 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.477173    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scpx8\" (UniqueName: \"kubernetes.io/projected/634277cb-f6c8-4254-b1aa-ccaebfe829a9-kube-api-access-scpx8\") pod \"multus-additional-cni-plugins-jwgfl\" (UID: \"634277cb-f6c8-4254-b1aa-ccaebfe829a9\") " pod="openshift-multus/multus-additional-cni-plugins-jwgfl"
Jun 15 05:48:06.571825 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.571729    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j6vx"
Jun 15 05:48:06.581947 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:06.581923    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb42bd55d_7f2c_4928_afa4_fc574974b8ae.slice/crio-056c50a71fb034ef0d9b03769e376052743382e2bd3b6ecada0bf257b5a74ef9 WatchSource:0}: Error finding container 056c50a71fb034ef0d9b03769e376052743382e2bd3b6ecada0bf257b5a74ef9: Status 404 returned error can't find the container with id 056c50a71fb034ef0d9b03769e376052743382e2bd3b6ecada0bf257b5a74ef9
Jun 15 05:48:06.587105 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.587087    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4dv8m"
Jun 15 05:48:06.594941 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:06.594920    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54e26677_5db1_4e31_94c1_a5d3c5db5047.slice/crio-13da2d41d4f60b1baef63e272e80b1ca1ff70f8732493aa99b220b05bc903adc WatchSource:0}: Error finding container 13da2d41d4f60b1baef63e272e80b1ca1ff70f8732493aa99b220b05bc903adc: Status 404 returned error can't find the container with id 13da2d41d4f60b1baef63e272e80b1ca1ff70f8732493aa99b220b05bc903adc
Jun 15 05:48:06.599916 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.599903    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jwgfl"
Jun 15 05:48:06.607622 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:06.607600    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod634277cb_f6c8_4254_b1aa_ccaebfe829a9.slice/crio-5769f6dd05f7457acb6425232d28fd0d218c8656aed3275daa07ced84039a1e3 WatchSource:0}: Error finding container 5769f6dd05f7457acb6425232d28fd0d218c8656aed3275daa07ced84039a1e3: Status 404 returned error can't find the container with id 5769f6dd05f7457acb6425232d28fd0d218c8656aed3275daa07ced84039a1e3
Jun 15 05:48:06.617096 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.617079    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ccxxl"
Jun 15 05:48:06.628324 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:06.628293    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd313c84b_5ccf_445d_bae0_2bc39eea2db2.slice/crio-59651bad616a08a44945a1ad03d16ca36effa9cd755a24ca99163c285022755d WatchSource:0}: Error finding container 59651bad616a08a44945a1ad03d16ca36effa9cd755a24ca99163c285022755d: Status 404 returned error can't find the container with id 59651bad616a08a44945a1ad03d16ca36effa9cd755a24ca99163c285022755d
Jun 15 05:48:06.640002 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.639976    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lhxv8"
Jun 15 05:48:06.646994 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.646971    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-kxfln"
Jun 15 05:48:06.649534 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:06.649492    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0313d892_f05a_4331_b972_c991f2ffe06a.slice/crio-8b866f6afd5063e9a5529c50ae26fe647731bdb331d26016aca4db16a59014b3 WatchSource:0}: Error finding container 8b866f6afd5063e9a5529c50ae26fe647731bdb331d26016aca4db16a59014b3: Status 404 returned error can't find the container with id 8b866f6afd5063e9a5529c50ae26fe647731bdb331d26016aca4db16a59014b3
Jun 15 05:48:06.651476 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.651451    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k4lpw"
Jun 15 05:48:06.658182 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.658163    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g9tgf"
Jun 15 05:48:06.658275 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:06.658201    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60d38c85_2ead_4253_915b_fa8462546084.slice/crio-ffc80572e2723e8d680408745d0d92cba85b213a5f89d222413c2e6625b7e069 WatchSource:0}: Error finding container ffc80572e2723e8d680408745d0d92cba85b213a5f89d222413c2e6625b7e069: Status 404 returned error can't find the container with id ffc80572e2723e8d680408745d0d92cba85b213a5f89d222413c2e6625b7e069
Jun 15 05:48:06.661884 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.661864    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:06.664912 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:06.664884    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48e95765_7c73_4f62_a381_bb60c8a04b1b.slice/crio-32d6b4998021b2303b2ac562af387fda041b736e3a1d1791b4bd961469d70b08 WatchSource:0}: Error finding container 32d6b4998021b2303b2ac562af387fda041b736e3a1d1791b4bd961469d70b08: Status 404 returned error can't find the container with id 32d6b4998021b2303b2ac562af387fda041b736e3a1d1791b4bd961469d70b08
Jun 15 05:48:06.676127 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:06.676102    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc80610bc_c188_476f_8a0c_ce642967f28e.slice/crio-c1dd18f839306f9c221e91b518de88503331ac9c92767f445d5b299765d174c4 WatchSource:0}: Error finding container c1dd18f839306f9c221e91b518de88503331ac9c92767f445d5b299765d174c4: Status 404 returned error can't find the container with id c1dd18f839306f9c221e91b518de88503331ac9c92767f445d5b299765d174c4
Jun 15 05:48:06.679159 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:06.679125    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5eacff_8b85_4b48_a883_22d29645244a.slice/crio-cd1422241ec22b56a4406e10b2213fdc4b490eef397b774f3391634f6560e24b WatchSource:0}: Error finding container cd1422241ec22b56a4406e10b2213fdc4b490eef397b774f3391634f6560e24b: Status 404 returned error can't find the container with id cd1422241ec22b56a4406e10b2213fdc4b490eef397b774f3391634f6560e24b
Jun 15 05:48:06.687725 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.687575    2572 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 15 05:48:06.863416 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.863339    2572 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 15 05:48:06.863837 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.863815    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c764213-65e7-4296-a8b6-023c581b36c1-original-pull-secret\") pod \"global-pull-secret-syncer-z9ws7\" (UID: \"8c764213-65e7-4296-a8b6-023c581b36c1\") " pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:06.863986 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:06.863970    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 15 05:48:06.864045 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:06.864035    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c764213-65e7-4296-a8b6-023c581b36c1-original-pull-secret podName:8c764213-65e7-4296-a8b6-023c581b36c1 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:07.864016817 +0000 UTC m=+3.012196801 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c764213-65e7-4296-a8b6-023c581b36c1-original-pull-secret") pod "global-pull-secret-syncer-z9ws7" (UID: "8c764213-65e7-4296-a8b6-023c581b36c1") : object "kube-system"/"original-pull-secret" not registered
Jun 15 05:48:06.964615 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:06.964577    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57565893-704b-4974-bc5b-c8f9cb932d15-metrics-certs\") pod \"network-metrics-daemon-vs8ck\" (UID: \"57565893-704b-4974-bc5b-c8f9cb932d15\") " pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:06.964805 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:06.964718    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:48:06.964805 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:06.964782    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57565893-704b-4974-bc5b-c8f9cb932d15-metrics-certs podName:57565893-704b-4974-bc5b-c8f9cb932d15 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:07.964763341 +0000 UTC m=+3.112943321 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57565893-704b-4974-bc5b-c8f9cb932d15-metrics-certs") pod "network-metrics-daemon-vs8ck" (UID: "57565893-704b-4974-bc5b-c8f9cb932d15") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:48:07.065436 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:07.065398    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mws52\" (UniqueName: \"kubernetes.io/projected/243c1d92-89f7-4ebb-838a-0f18bd3216d8-kube-api-access-mws52\") pod \"network-check-target-9b7jj\" (UID: \"243c1d92-89f7-4ebb-838a-0f18bd3216d8\") " pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:07.065611 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:07.065570    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 15 05:48:07.065611 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:07.065593    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 15 05:48:07.065611 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:07.065606    2572 projected.go:196] Error preparing data for projected volume kube-api-access-mws52 for pod openshift-network-diagnostics/network-check-target-9b7jj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:48:07.065869 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:07.065665    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/243c1d92-89f7-4ebb-838a-0f18bd3216d8-kube-api-access-mws52 podName:243c1d92-89f7-4ebb-838a-0f18bd3216d8 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:08.065647181 +0000 UTC m=+3.213827161 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mws52" (UniqueName: "kubernetes.io/projected/243c1d92-89f7-4ebb-838a-0f18bd3216d8-kube-api-access-mws52") pod "network-check-target-9b7jj" (UID: "243c1d92-89f7-4ebb-838a-0f18bd3216d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:48:07.292970 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:07.292933    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-14 05:43:06 +0000 UTC" deadline="2027-11-15 13:14:03.074076337 +0000 UTC"
Jun 15 05:48:07.292970 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:07.292969    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12439h25m55.78111209s"
Jun 15 05:48:07.358762 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:07.358699    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g9tgf" event={"ID":"c80610bc-c188-476f-8a0c-ce642967f28e","Type":"ContainerStarted","Data":"c1dd18f839306f9c221e91b518de88503331ac9c92767f445d5b299765d174c4"}
Jun 15 05:48:07.379819 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:07.379643    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k4lpw" event={"ID":"48e95765-7c73-4f62-a381-bb60c8a04b1b","Type":"ContainerStarted","Data":"32d6b4998021b2303b2ac562af387fda041b736e3a1d1791b4bd961469d70b08"}
Jun 15 05:48:07.394271 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:07.394181    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kxfln" event={"ID":"60d38c85-2ead-4253-915b-fa8462546084","Type":"ContainerStarted","Data":"ffc80572e2723e8d680408745d0d92cba85b213a5f89d222413c2e6625b7e069"}
Jun 15 05:48:07.402179 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:07.402110    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lhxv8" event={"ID":"0313d892-f05a-4331-b972-c991f2ffe06a","Type":"ContainerStarted","Data":"8b866f6afd5063e9a5529c50ae26fe647731bdb331d26016aca4db16a59014b3"}
Jun 15 05:48:07.410441 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:07.410342    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ccxxl" event={"ID":"d313c84b-5ccf-445d-bae0-2bc39eea2db2","Type":"ContainerStarted","Data":"59651bad616a08a44945a1ad03d16ca36effa9cd755a24ca99163c285022755d"}
Jun 15 05:48:07.434048 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:07.434021    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jwgfl" event={"ID":"634277cb-f6c8-4254-b1aa-ccaebfe829a9","Type":"ContainerStarted","Data":"5769f6dd05f7457acb6425232d28fd0d218c8656aed3275daa07ced84039a1e3"}
Jun 15 05:48:07.441282 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:07.441251    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j6vx" event={"ID":"b42bd55d-7f2c-4928-afa4-fc574974b8ae","Type":"ContainerStarted","Data":"056c50a71fb034ef0d9b03769e376052743382e2bd3b6ecada0bf257b5a74ef9"}
Jun 15 05:48:07.447057 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:07.446834    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-98qt9" event={"ID":"0b5eacff-8b85-4b48-a883-22d29645244a","Type":"ContainerStarted","Data":"cd1422241ec22b56a4406e10b2213fdc4b490eef397b774f3391634f6560e24b"}
Jun 15 05:48:07.472591 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:07.472433    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4dv8m" event={"ID":"54e26677-5db1-4e31-94c1-a5d3c5db5047","Type":"ContainerStarted","Data":"13da2d41d4f60b1baef63e272e80b1ca1ff70f8732493aa99b220b05bc903adc"}
Jun 15 05:48:07.483628 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:07.483546    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-16.ec2.internal" event={"ID":"22301900658e933839cf55f1e43739b1","Type":"ContainerStarted","Data":"cba24e8a944061f6c72770dd5b3d16ecce3f0e42f5193980bb0e8201172ef6a7"}
Jun 15 05:48:07.505372 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:07.505291    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-16.ec2.internal" event={"ID":"f55c91d0f7b296b874c42e3338400df2","Type":"ContainerStarted","Data":"43e17a5dcfff39deb49858189c7677dd2082e66e2f85b9f7a87142338e32c69c"}
Jun 15 05:48:07.877325 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:07.877287    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c764213-65e7-4296-a8b6-023c581b36c1-original-pull-secret\") pod \"global-pull-secret-syncer-z9ws7\" (UID: \"8c764213-65e7-4296-a8b6-023c581b36c1\") " pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:07.877506 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:07.877491    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 15 05:48:07.877576 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:07.877561    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c764213-65e7-4296-a8b6-023c581b36c1-original-pull-secret podName:8c764213-65e7-4296-a8b6-023c581b36c1 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:09.877541517 +0000 UTC m=+5.025721496 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c764213-65e7-4296-a8b6-023c581b36c1-original-pull-secret") pod "global-pull-secret-syncer-z9ws7" (UID: "8c764213-65e7-4296-a8b6-023c581b36c1") : object "kube-system"/"original-pull-secret" not registered
Jun 15 05:48:07.978332 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:07.978293    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57565893-704b-4974-bc5b-c8f9cb932d15-metrics-certs\") pod \"network-metrics-daemon-vs8ck\" (UID: \"57565893-704b-4974-bc5b-c8f9cb932d15\") " pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:07.978532 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:07.978460    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:48:07.978532 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:07.978529    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57565893-704b-4974-bc5b-c8f9cb932d15-metrics-certs podName:57565893-704b-4974-bc5b-c8f9cb932d15 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:09.978509061 +0000 UTC m=+5.126689041 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57565893-704b-4974-bc5b-c8f9cb932d15-metrics-certs") pod "network-metrics-daemon-vs8ck" (UID: "57565893-704b-4974-bc5b-c8f9cb932d15") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:48:08.078993 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:08.078956    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mws52\" (UniqueName: \"kubernetes.io/projected/243c1d92-89f7-4ebb-838a-0f18bd3216d8-kube-api-access-mws52\") pod \"network-check-target-9b7jj\" (UID: \"243c1d92-89f7-4ebb-838a-0f18bd3216d8\") " pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:08.079170 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:08.079153    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 15 05:48:08.079245 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:08.079177    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 15 05:48:08.079245 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:08.079189    2572 projected.go:196] Error preparing data for projected volume kube-api-access-mws52 for pod openshift-network-diagnostics/network-check-target-9b7jj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:48:08.079338 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:08.079248    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/243c1d92-89f7-4ebb-838a-0f18bd3216d8-kube-api-access-mws52 podName:243c1d92-89f7-4ebb-838a-0f18bd3216d8 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:10.079230498 +0000 UTC m=+5.227410479 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mws52" (UniqueName: "kubernetes.io/projected/243c1d92-89f7-4ebb-838a-0f18bd3216d8-kube-api-access-mws52") pod "network-check-target-9b7jj" (UID: "243c1d92-89f7-4ebb-838a-0f18bd3216d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:48:08.294087 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:08.293979    2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-06-14 05:43:06 +0000 UTC" deadline="2028-03-28 23:43:38.030535948 +0000 UTC"
Jun 15 05:48:08.294087 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:08.294035    2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15665h55m29.736504644s"
Jun 15 05:48:08.319517 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:08.318713    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:08.319517 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:08.318840    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9ws7" podUID="8c764213-65e7-4296-a8b6-023c581b36c1"
Jun 15 05:48:08.319517 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:08.319235    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:08.319517 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:08.319332    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vs8ck" podUID="57565893-704b-4974-bc5b-c8f9cb932d15"
Jun 15 05:48:08.319517 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:08.319414    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:08.319517 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:08.319480    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b7jj" podUID="243c1d92-89f7-4ebb-838a-0f18bd3216d8"
Jun 15 05:48:08.667114 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:08.667084    2572 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 15 05:48:09.119856 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:09.119765    2572 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jun 15 05:48:09.894148 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:09.894115    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c764213-65e7-4296-a8b6-023c581b36c1-original-pull-secret\") pod \"global-pull-secret-syncer-z9ws7\" (UID: \"8c764213-65e7-4296-a8b6-023c581b36c1\") " pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:09.894639 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:09.894258    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 15 05:48:09.894639 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:09.894310    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c764213-65e7-4296-a8b6-023c581b36c1-original-pull-secret podName:8c764213-65e7-4296-a8b6-023c581b36c1 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:13.894296871 +0000 UTC m=+9.042476976 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c764213-65e7-4296-a8b6-023c581b36c1-original-pull-secret") pod "global-pull-secret-syncer-z9ws7" (UID: "8c764213-65e7-4296-a8b6-023c581b36c1") : object "kube-system"/"original-pull-secret" not registered
Jun 15 05:48:09.994635 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:09.994543    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57565893-704b-4974-bc5b-c8f9cb932d15-metrics-certs\") pod \"network-metrics-daemon-vs8ck\" (UID: \"57565893-704b-4974-bc5b-c8f9cb932d15\") " pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:09.994805 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:09.994666    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:48:09.994805 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:09.994736    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57565893-704b-4974-bc5b-c8f9cb932d15-metrics-certs podName:57565893-704b-4974-bc5b-c8f9cb932d15 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:13.994718367 +0000 UTC m=+9.142898348 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57565893-704b-4974-bc5b-c8f9cb932d15-metrics-certs") pod "network-metrics-daemon-vs8ck" (UID: "57565893-704b-4974-bc5b-c8f9cb932d15") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:48:10.096380 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:10.095766    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mws52\" (UniqueName: \"kubernetes.io/projected/243c1d92-89f7-4ebb-838a-0f18bd3216d8-kube-api-access-mws52\") pod \"network-check-target-9b7jj\" (UID: \"243c1d92-89f7-4ebb-838a-0f18bd3216d8\") " pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:10.096380 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:10.095944    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 15 05:48:10.096380 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:10.095965    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 15 05:48:10.096380 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:10.095978    2572 projected.go:196] Error preparing data for projected volume kube-api-access-mws52 for pod openshift-network-diagnostics/network-check-target-9b7jj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:48:10.096380 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:10.096038    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/243c1d92-89f7-4ebb-838a-0f18bd3216d8-kube-api-access-mws52 podName:243c1d92-89f7-4ebb-838a-0f18bd3216d8 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:14.096019755 +0000 UTC m=+9.244199748 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mws52" (UniqueName: "kubernetes.io/projected/243c1d92-89f7-4ebb-838a-0f18bd3216d8-kube-api-access-mws52") pod "network-check-target-9b7jj" (UID: "243c1d92-89f7-4ebb-838a-0f18bd3216d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:48:10.318881 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:10.318810    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:10.319027 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:10.318934    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b7jj" podUID="243c1d92-89f7-4ebb-838a-0f18bd3216d8"
Jun 15 05:48:10.319027 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:10.318997    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:10.319165 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:10.319090    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9ws7" podUID="8c764213-65e7-4296-a8b6-023c581b36c1"
Jun 15 05:48:10.319165 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:10.319133    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:10.319313 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:10.319208    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vs8ck" podUID="57565893-704b-4974-bc5b-c8f9cb932d15"
Jun 15 05:48:12.318260 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:12.318220    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:12.318739 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:12.318259    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:12.318739 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:12.318342    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9ws7" podUID="8c764213-65e7-4296-a8b6-023c581b36c1"
Jun 15 05:48:12.318739 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:12.318433    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b7jj" podUID="243c1d92-89f7-4ebb-838a-0f18bd3216d8"
Jun 15 05:48:12.318739 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:12.318223    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:12.318739 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:12.318566    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vs8ck" podUID="57565893-704b-4974-bc5b-c8f9cb932d15"
Jun 15 05:48:13.929907 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:13.929866    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c764213-65e7-4296-a8b6-023c581b36c1-original-pull-secret\") pod \"global-pull-secret-syncer-z9ws7\" (UID: \"8c764213-65e7-4296-a8b6-023c581b36c1\") " pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:13.930383 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:13.930020    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 15 05:48:13.930383 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:13.930096    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c764213-65e7-4296-a8b6-023c581b36c1-original-pull-secret podName:8c764213-65e7-4296-a8b6-023c581b36c1 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:21.930068971 +0000 UTC m=+17.078248973 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c764213-65e7-4296-a8b6-023c581b36c1-original-pull-secret") pod "global-pull-secret-syncer-z9ws7" (UID: "8c764213-65e7-4296-a8b6-023c581b36c1") : object "kube-system"/"original-pull-secret" not registered
Jun 15 05:48:14.030797 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:14.030758    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57565893-704b-4974-bc5b-c8f9cb932d15-metrics-certs\") pod \"network-metrics-daemon-vs8ck\" (UID: \"57565893-704b-4974-bc5b-c8f9cb932d15\") " pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:14.030971 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:14.030934    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:48:14.031029 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:14.031008    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57565893-704b-4974-bc5b-c8f9cb932d15-metrics-certs podName:57565893-704b-4974-bc5b-c8f9cb932d15 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:22.030987101 +0000 UTC m=+17.179167079 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57565893-704b-4974-bc5b-c8f9cb932d15-metrics-certs") pod "network-metrics-daemon-vs8ck" (UID: "57565893-704b-4974-bc5b-c8f9cb932d15") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:48:14.131910 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:14.131871    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mws52\" (UniqueName: \"kubernetes.io/projected/243c1d92-89f7-4ebb-838a-0f18bd3216d8-kube-api-access-mws52\") pod \"network-check-target-9b7jj\" (UID: \"243c1d92-89f7-4ebb-838a-0f18bd3216d8\") " pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:14.132094 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:14.132038    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 15 05:48:14.132094 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:14.132056    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 15 05:48:14.132094 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:14.132069    2572 projected.go:196] Error preparing data for projected volume kube-api-access-mws52 for pod openshift-network-diagnostics/network-check-target-9b7jj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:48:14.132248 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:14.132124    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/243c1d92-89f7-4ebb-838a-0f18bd3216d8-kube-api-access-mws52 podName:243c1d92-89f7-4ebb-838a-0f18bd3216d8 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:22.132106018 +0000 UTC m=+17.280285997 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mws52" (UniqueName: "kubernetes.io/projected/243c1d92-89f7-4ebb-838a-0f18bd3216d8-kube-api-access-mws52") pod "network-check-target-9b7jj" (UID: "243c1d92-89f7-4ebb-838a-0f18bd3216d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:48:14.318956 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:14.318858    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:14.319127 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:14.318993    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:14.319127 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:14.319009    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b7jj" podUID="243c1d92-89f7-4ebb-838a-0f18bd3216d8"
Jun 15 05:48:14.319247 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:14.319122    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vs8ck" podUID="57565893-704b-4974-bc5b-c8f9cb932d15"
Jun 15 05:48:14.319247 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:14.319180    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:14.319345 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:14.319252    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9ws7" podUID="8c764213-65e7-4296-a8b6-023c581b36c1"
Jun 15 05:48:16.318594 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:16.318538    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:16.318594 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:16.318578    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:16.319175 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:16.318578    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:16.319175 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:16.318664    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b7jj" podUID="243c1d92-89f7-4ebb-838a-0f18bd3216d8"
Jun 15 05:48:16.319175 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:16.318768    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vs8ck" podUID="57565893-704b-4974-bc5b-c8f9cb932d15"
Jun 15 05:48:16.319175 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:16.318866    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9ws7" podUID="8c764213-65e7-4296-a8b6-023c581b36c1"
Jun 15 05:48:18.319227 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:18.319133    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:18.319227 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:18.319175    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:18.319779 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:18.319248    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:18.319779 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:18.319263    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b7jj" podUID="243c1d92-89f7-4ebb-838a-0f18bd3216d8"
Jun 15 05:48:18.319779 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:18.319328    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9ws7" podUID="8c764213-65e7-4296-a8b6-023c581b36c1"
Jun 15 05:48:18.319779 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:18.319436    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vs8ck" podUID="57565893-704b-4974-bc5b-c8f9cb932d15"
Jun 15 05:48:20.318432 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:20.318403    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:20.318432 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:20.318430    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:20.318946 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:20.318442    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:20.318946 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:20.318535    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b7jj" podUID="243c1d92-89f7-4ebb-838a-0f18bd3216d8"
Jun 15 05:48:20.318946 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:20.318662    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9ws7" podUID="8c764213-65e7-4296-a8b6-023c581b36c1"
Jun 15 05:48:20.318946 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:20.318759    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vs8ck" podUID="57565893-704b-4974-bc5b-c8f9cb932d15"
Jun 15 05:48:21.991412 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:21.991378    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c764213-65e7-4296-a8b6-023c581b36c1-original-pull-secret\") pod \"global-pull-secret-syncer-z9ws7\" (UID: \"8c764213-65e7-4296-a8b6-023c581b36c1\") " pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:21.991902 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:21.991554    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 15 05:48:21.991902 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:21.991622    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c764213-65e7-4296-a8b6-023c581b36c1-original-pull-secret podName:8c764213-65e7-4296-a8b6-023c581b36c1 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:37.991607386 +0000 UTC m=+33.139787361 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c764213-65e7-4296-a8b6-023c581b36c1-original-pull-secret") pod "global-pull-secret-syncer-z9ws7" (UID: "8c764213-65e7-4296-a8b6-023c581b36c1") : object "kube-system"/"original-pull-secret" not registered
Jun 15 05:48:22.092636 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:22.092599    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57565893-704b-4974-bc5b-c8f9cb932d15-metrics-certs\") pod \"network-metrics-daemon-vs8ck\" (UID: \"57565893-704b-4974-bc5b-c8f9cb932d15\") " pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:22.092781 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:22.092749    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:48:22.092839 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:22.092808    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57565893-704b-4974-bc5b-c8f9cb932d15-metrics-certs podName:57565893-704b-4974-bc5b-c8f9cb932d15 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:38.092793581 +0000 UTC m=+33.240973556 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57565893-704b-4974-bc5b-c8f9cb932d15-metrics-certs") pod "network-metrics-daemon-vs8ck" (UID: "57565893-704b-4974-bc5b-c8f9cb932d15") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:48:22.193786 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:22.193751    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mws52\" (UniqueName: \"kubernetes.io/projected/243c1d92-89f7-4ebb-838a-0f18bd3216d8-kube-api-access-mws52\") pod \"network-check-target-9b7jj\" (UID: \"243c1d92-89f7-4ebb-838a-0f18bd3216d8\") " pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:22.193934 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:22.193884    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 15 05:48:22.193934 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:22.193900    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 15 05:48:22.193934 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:22.193908    2572 projected.go:196] Error preparing data for projected volume kube-api-access-mws52 for pod openshift-network-diagnostics/network-check-target-9b7jj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:48:22.194048 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:22.193993    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/243c1d92-89f7-4ebb-838a-0f18bd3216d8-kube-api-access-mws52 podName:243c1d92-89f7-4ebb-838a-0f18bd3216d8 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:38.193974639 +0000 UTC m=+33.342154616 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mws52" (UniqueName: "kubernetes.io/projected/243c1d92-89f7-4ebb-838a-0f18bd3216d8-kube-api-access-mws52") pod "network-check-target-9b7jj" (UID: "243c1d92-89f7-4ebb-838a-0f18bd3216d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:48:22.318768 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:22.318679    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:22.318768 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:22.318697    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:22.318768 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:22.318679    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:22.319017 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:22.318821    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9ws7" podUID="8c764213-65e7-4296-a8b6-023c581b36c1"
Jun 15 05:48:22.319017 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:22.318882    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vs8ck" podUID="57565893-704b-4974-bc5b-c8f9cb932d15"
Jun 15 05:48:22.319017 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:22.318959    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b7jj" podUID="243c1d92-89f7-4ebb-838a-0f18bd3216d8"
Jun 15 05:48:24.318978 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:24.318945    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:24.319459 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:24.319073    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:24.319459 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:24.319200    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vs8ck" podUID="57565893-704b-4974-bc5b-c8f9cb932d15"
Jun 15 05:48:24.319459 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:24.319077    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:24.319459 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:24.319075    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9ws7" podUID="8c764213-65e7-4296-a8b6-023c581b36c1"
Jun 15 05:48:24.319459 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:24.319339    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b7jj" podUID="243c1d92-89f7-4ebb-838a-0f18bd3216d8"
Jun 15 05:48:26.318600 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:26.318576    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:26.318925 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:26.318577    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:26.318925 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:26.318689    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vs8ck" podUID="57565893-704b-4974-bc5b-c8f9cb932d15"
Jun 15 05:48:26.318925 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:26.318744    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b7jj" podUID="243c1d92-89f7-4ebb-838a-0f18bd3216d8"
Jun 15 05:48:26.318925 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:26.318577    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:26.318925 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:26.318847    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9ws7" podUID="8c764213-65e7-4296-a8b6-023c581b36c1"
Jun 15 05:48:26.557853 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:26.557650    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-16.ec2.internal" event={"ID":"f55c91d0f7b296b874c42e3338400df2","Type":"ContainerStarted","Data":"bf09a38e491afe60de56c34d6a9d878b1aa476141e3a5ecf1aacfaf670cc8e99"}
Jun 15 05:48:26.559372 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:26.559295    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k4lpw" event={"ID":"48e95765-7c73-4f62-a381-bb60c8a04b1b","Type":"ContainerStarted","Data":"3633aa68d68c1fb45ec6a134e1ee5b3061e2611c82ed29b04f4be3b27a63b724"}
Jun 15 05:48:26.560641 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:26.560621    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ccxxl" event={"ID":"d313c84b-5ccf-445d-bae0-2bc39eea2db2","Type":"ContainerStarted","Data":"ec6bd388675cdbe13347f5dbc1f1377d28f1e73f7c9319bf7f4a7db3214985b1"}
Jun 15 05:48:26.574018 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:26.573964    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-16.ec2.internal" podStartSLOduration=20.573947005 podStartE2EDuration="20.573947005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:26.573328429 +0000 UTC m=+21.721508427" watchObservedRunningTime="2026-06-15 05:48:26.573947005 +0000 UTC m=+21.722127003"
Jun 15 05:48:26.619771 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:26.619720    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-ccxxl" podStartSLOduration=1.974333656 podStartE2EDuration="21.619697768s" totalImagesPullingTime="19.645364112s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:48:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:26.619562717 +0000 UTC m=+21.767742715" watchObservedRunningTime="2026-06-15 05:48:26.619697768 +0000 UTC m=+21.767877765"
Jun 15 05:48:26.620187 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:26.620153    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-k4lpw" podStartSLOduration=2.044207928 podStartE2EDuration="21.620144319s" totalImagesPullingTime="19.575936391s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:48:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:26.597634381 +0000 UTC m=+21.745814375" watchObservedRunningTime="2026-06-15 05:48:26.620144319 +0000 UTC m=+21.768324318"
Jun 15 05:48:27.565804 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:27.565523    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lhxv8" event={"ID":"0313d892-f05a-4331-b972-c991f2ffe06a","Type":"ContainerStarted","Data":"ac4716a31972e503f362ef6ee47930e3316214d14fb2cd39d88efd34c22514b8"}
Jun 15 05:48:27.570178 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:27.570095    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 05:48:27.571438 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:27.570622    2572 generic.go:356] "Generic (PLEG): container finished" podID="0b5eacff-8b85-4b48-a883-22d29645244a" containerID="fd360f98436042f45f54f08f535d31a6113be46fba2d6b44cd94498e4c5f8317" exitCode=1
Jun 15 05:48:27.571438 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:27.570671    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-98qt9" event={"ID":"0b5eacff-8b85-4b48-a883-22d29645244a","Type":"ContainerStarted","Data":"18903ce5d010a8ad3fd28a6134dc83b765cc2b5a0808ba0f97d4230dd5f18350"}
Jun 15 05:48:27.571438 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:27.570755    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-98qt9" event={"ID":"0b5eacff-8b85-4b48-a883-22d29645244a","Type":"ContainerStarted","Data":"f013b216c801b2a36dc76e5a01c1697a025249224303fea8a469be797c18aa4e"}
Jun 15 05:48:27.571438 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:27.570802    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-98qt9" event={"ID":"0b5eacff-8b85-4b48-a883-22d29645244a","Type":"ContainerStarted","Data":"856ac6c2f6b67d48140c432389b64a5dad3521d7eaa248ae9c3d88fa4b604e54"}
Jun 15 05:48:27.571438 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:27.570848    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-98qt9" event={"ID":"0b5eacff-8b85-4b48-a883-22d29645244a","Type":"ContainerStarted","Data":"1c2ab1d320bfad7fc61b5419ca4c503b481f92f730da1b9dbb80dfe60194fd61"}
Jun 15 05:48:27.571438 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:27.570893    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-98qt9" event={"ID":"0b5eacff-8b85-4b48-a883-22d29645244a","Type":"ContainerDied","Data":"fd360f98436042f45f54f08f535d31a6113be46fba2d6b44cd94498e4c5f8317"}
Jun 15 05:48:27.571438 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:27.570940    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-98qt9" event={"ID":"0b5eacff-8b85-4b48-a883-22d29645244a","Type":"ContainerStarted","Data":"04ffa539373716d45d524008ef81665dc5c4eaf773c29ebd6df8569ecd905e03"}
Jun 15 05:48:28.318834 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:28.318801    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:28.319006 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:28.318875    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:28.319006 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:28.318971    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:28.319006 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:28.318992    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b7jj" podUID="243c1d92-89f7-4ebb-838a-0f18bd3216d8"
Jun 15 05:48:28.319116 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:28.319054    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9ws7" podUID="8c764213-65e7-4296-a8b6-023c581b36c1"
Jun 15 05:48:28.319157 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:28.319124    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vs8ck" podUID="57565893-704b-4974-bc5b-c8f9cb932d15"
Jun 15 05:48:28.574278 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:28.574200    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4dv8m" event={"ID":"54e26677-5db1-4e31-94c1-a5d3c5db5047","Type":"ContainerStarted","Data":"a97e13c0aab7ea4b34160c9f832ff155acfa473e08055b2a0ba3d135fc0275fc"}
Jun 15 05:48:28.575608 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:28.575584    2572 generic.go:356] "Generic (PLEG): container finished" podID="22301900658e933839cf55f1e43739b1" containerID="24727ec256ff7a12e858d9594d4db70f5d0205b2ffc3cbd8942e2e0834dda2d3" exitCode=0
Jun 15 05:48:28.575680 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:28.575659    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-16.ec2.internal" event={"ID":"22301900658e933839cf55f1e43739b1","Type":"ContainerDied","Data":"24727ec256ff7a12e858d9594d4db70f5d0205b2ffc3cbd8942e2e0834dda2d3"}
Jun 15 05:48:28.577022 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:28.577001    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g9tgf" event={"ID":"c80610bc-c188-476f-8a0c-ce642967f28e","Type":"ContainerStarted","Data":"c0418e3bbcb2a5eaa0e849534750977e22acec1778c15b16b156e45eaa767f03"}
Jun 15 05:48:28.578480 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:28.578457    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-kxfln" event={"ID":"60d38c85-2ead-4253-915b-fa8462546084","Type":"ContainerStarted","Data":"411ce97dd8022f99a16e102635a4d0f09a91f5237481f8aaa83577272a2242e2"}
Jun 15 05:48:28.579785 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:28.579767    2572 generic.go:356] "Generic (PLEG): container finished" podID="634277cb-f6c8-4254-b1aa-ccaebfe829a9" containerID="cebc68409910b85ad3b8ed8b946c6c240b62f985549b5b88cb0cfcc0cbf87e25" exitCode=0
Jun 15 05:48:28.579865 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:28.579827    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jwgfl" event={"ID":"634277cb-f6c8-4254-b1aa-ccaebfe829a9","Type":"ContainerDied","Data":"cebc68409910b85ad3b8ed8b946c6c240b62f985549b5b88cb0cfcc0cbf87e25"}
Jun 15 05:48:28.581309 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:28.581289    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j6vx" event={"ID":"b42bd55d-7f2c-4928-afa4-fc574974b8ae","Type":"ContainerStarted","Data":"8dc5b2eb04f200ac1e99acd94ff9364ebc7b1be0f41df021fb8ff798ef3fdbca"}
Jun 15 05:48:28.595336 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:28.595297    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4dv8m" podStartSLOduration=3.951273411 podStartE2EDuration="23.595286225s" totalImagesPullingTime="19.644012814s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:48:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:28.595189006 +0000 UTC m=+23.743369004" watchObservedRunningTime="2026-06-15 05:48:28.595286225 +0000 UTC m=+23.743466222"
Jun 15 05:48:28.610222 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:28.610183    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-kxfln" podStartSLOduration=4.027732616 podStartE2EDuration="23.610168028s" totalImagesPullingTime="19.582435412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:48:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:28.60975661 +0000 UTC m=+23.757936607" watchObservedRunningTime="2026-06-15 05:48:28.610168028 +0000 UTC m=+23.758348025"
Jun 15 05:48:28.650352 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:28.650310    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lhxv8" podStartSLOduration=4.060371702 podStartE2EDuration="23.65029733s" totalImagesPullingTime="19.589925628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:48:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:28.650209902 +0000 UTC m=+23.798389897" watchObservedRunningTime="2026-06-15 05:48:28.65029733 +0000 UTC m=+23.798477326"
Jun 15 05:48:28.666614 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:28.666574    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-g9tgf" podStartSLOduration=4.111469315 podStartE2EDuration="23.666560432s" totalImagesPullingTime="19.555091117s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:48:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:28.666111922 +0000 UTC m=+23.814291919" watchObservedRunningTime="2026-06-15 05:48:28.666560432 +0000 UTC m=+23.814740428"
Jun 15 05:48:28.850706 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:28.850567    2572 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jun 15 05:48:29.202018 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:29.201982    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-kxfln"
Jun 15 05:48:29.202605 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:29.202588    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-kxfln"
Jun 15 05:48:29.328121 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:29.328023    2572 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-06-15T05:48:28.850703807Z","UUID":"7efa0f44-9e01-4601-a130-e8df884ac114","Handler":null,"Name":"","Endpoint":""}
Jun 15 05:48:29.331490 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:29.331463    2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jun 15 05:48:29.331641 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:29.331509    2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jun 15 05:48:29.586435 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:29.586344    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j6vx" event={"ID":"b42bd55d-7f2c-4928-afa4-fc574974b8ae","Type":"ContainerStarted","Data":"866ee318b164e9d9d259d87e8f5888694da863b7a798473744b109ecfa89d42d"}
Jun 15 05:48:29.590067 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:29.590040    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 05:48:29.590561 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:29.590534    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-98qt9" event={"ID":"0b5eacff-8b85-4b48-a883-22d29645244a","Type":"ContainerStarted","Data":"36d96e6dcce574aa2f70482deecfe566ee28cb039ab7aeaa612ed7c0b1714844"}
Jun 15 05:48:29.592565 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:29.592540    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-16.ec2.internal" event={"ID":"22301900658e933839cf55f1e43739b1","Type":"ContainerStarted","Data":"59d9270733400070d4a5b5d9df646e0c4ce0aaf1dc6ea3afaf5e4001bbf3841c"}
Jun 15 05:48:29.593234 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:29.592912    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-kxfln"
Jun 15 05:48:29.593350 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:29.593329    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-kxfln"
Jun 15 05:48:29.611268 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:29.611213    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-16.ec2.internal" podStartSLOduration=23.611195575 podStartE2EDuration="23.611195575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:29.610932779 +0000 UTC m=+24.759112791" watchObservedRunningTime="2026-06-15 05:48:29.611195575 +0000 UTC m=+24.759375577"
Jun 15 05:48:30.319112 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:30.319080    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:30.319409 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:30.319080    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:30.319409 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:30.319189    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b7jj" podUID="243c1d92-89f7-4ebb-838a-0f18bd3216d8"
Jun 15 05:48:30.319409 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:30.319081    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:30.319409 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:30.319243    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vs8ck" podUID="57565893-704b-4974-bc5b-c8f9cb932d15"
Jun 15 05:48:30.319409 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:30.319371    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9ws7" podUID="8c764213-65e7-4296-a8b6-023c581b36c1"
Jun 15 05:48:30.596743 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:30.596651    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j6vx" event={"ID":"b42bd55d-7f2c-4928-afa4-fc574974b8ae","Type":"ContainerStarted","Data":"9b01b0c8925e42e51ce159a6370710662b1128bba9ce6cc71405ca51c15825cb"}
Jun 15 05:48:32.318208 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:32.318175    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:32.318757 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:32.318288    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:32.318757 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:32.318296    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b7jj" podUID="243c1d92-89f7-4ebb-838a-0f18bd3216d8"
Jun 15 05:48:32.318757 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:32.318337    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:32.318757 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:32.318437    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9ws7" podUID="8c764213-65e7-4296-a8b6-023c581b36c1"
Jun 15 05:48:32.318757 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:32.318525    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vs8ck" podUID="57565893-704b-4974-bc5b-c8f9cb932d15"
Jun 15 05:48:33.603853 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:33.603649    2572 generic.go:356] "Generic (PLEG): container finished" podID="634277cb-f6c8-4254-b1aa-ccaebfe829a9" containerID="e1ee7f36d7ab6e2847b5f3770fce57da02e0fb0ce04766c38e8b0baa3396db9f" exitCode=0
Jun 15 05:48:33.604314 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:33.603735    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jwgfl" event={"ID":"634277cb-f6c8-4254-b1aa-ccaebfe829a9","Type":"ContainerDied","Data":"e1ee7f36d7ab6e2847b5f3770fce57da02e0fb0ce04766c38e8b0baa3396db9f"}
Jun 15 05:48:33.607399 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:33.607383    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 05:48:33.607757 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:33.607737    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-98qt9" event={"ID":"0b5eacff-8b85-4b48-a883-22d29645244a","Type":"ContainerStarted","Data":"b828d853c7c791b6424393a7d317ac391edb80481e7ca76d4ceec297f7970d95"}
Jun 15 05:48:33.608006 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:33.607987    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:33.608102 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:33.608064    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:33.608162 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:33.608138    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:33.608162 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:33.608149    2572 scope.go:117] "RemoveContainer" containerID="fd360f98436042f45f54f08f535d31a6113be46fba2d6b44cd94498e4c5f8317"
Jun 15 05:48:33.625087 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:33.625067    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:33.625176 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:33.625168    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:48:33.628495 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:33.628447    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2j6vx" podStartSLOduration=6.871730254 podStartE2EDuration="28.628429176s" totalImagesPullingTime="21.756698922s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:48:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:30.62186484 +0000 UTC m=+25.770044836" watchObservedRunningTime="2026-06-15 05:48:33.628429176 +0000 UTC m=+28.776609185"
Jun 15 05:48:34.318947 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:34.318906    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:34.319191 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:34.319010    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:34.319191 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:34.319017    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b7jj" podUID="243c1d92-89f7-4ebb-838a-0f18bd3216d8"
Jun 15 05:48:34.319191 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:34.319040    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:34.319191 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:34.319120    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vs8ck" podUID="57565893-704b-4974-bc5b-c8f9cb932d15"
Jun 15 05:48:34.319191 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:34.319187    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9ws7" podUID="8c764213-65e7-4296-a8b6-023c581b36c1"
Jun 15 05:48:34.613228 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:34.613160    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 05:48:34.613607 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:34.613479    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-98qt9" event={"ID":"0b5eacff-8b85-4b48-a883-22d29645244a","Type":"ContainerStarted","Data":"0e6aea4a21bf8b63253b73d4766dbe040d8c5607333029db4ce441e3b38fae4a"}
Jun 15 05:48:34.642235 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:34.642193    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-98qt9" podStartSLOduration=9.738070124 podStartE2EDuration="29.642178689s" totalImagesPullingTime="19.904108565s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:48:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:34.640897721 +0000 UTC m=+29.789077718" watchObservedRunningTime="2026-06-15 05:48:34.642178689 +0000 UTC m=+29.790358690"
Jun 15 05:48:35.617779 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:35.617746    2572 generic.go:356] "Generic (PLEG): container finished" podID="634277cb-f6c8-4254-b1aa-ccaebfe829a9" containerID="b633403a12c9c3586a3b7114e2b2649adb1119f90965ad9fc59e3a57f010c0a7" exitCode=0
Jun 15 05:48:35.618207 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:35.617822    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jwgfl" event={"ID":"634277cb-f6c8-4254-b1aa-ccaebfe829a9","Type":"ContainerDied","Data":"b633403a12c9c3586a3b7114e2b2649adb1119f90965ad9fc59e3a57f010c0a7"}
Jun 15 05:48:36.318853 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:36.318820    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:36.319021 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:36.318820    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:36.319021 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:36.318928    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vs8ck" podUID="57565893-704b-4974-bc5b-c8f9cb932d15"
Jun 15 05:48:36.319021 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:36.318820    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:36.319166 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:36.318992    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b7jj" podUID="243c1d92-89f7-4ebb-838a-0f18bd3216d8"
Jun 15 05:48:36.319166 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:36.319100    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9ws7" podUID="8c764213-65e7-4296-a8b6-023c581b36c1"
Jun 15 05:48:37.624793 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:37.624607    2572 generic.go:356] "Generic (PLEG): container finished" podID="634277cb-f6c8-4254-b1aa-ccaebfe829a9" containerID="5395fe1d0d3f33cf6881969a9641b3f08e71a33478c95020c72f3876f75f8a89" exitCode=0
Jun 15 05:48:37.625205 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:37.624681    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jwgfl" event={"ID":"634277cb-f6c8-4254-b1aa-ccaebfe829a9","Type":"ContainerDied","Data":"5395fe1d0d3f33cf6881969a9641b3f08e71a33478c95020c72f3876f75f8a89"}
Jun 15 05:48:38.019889 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:38.019861    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c764213-65e7-4296-a8b6-023c581b36c1-original-pull-secret\") pod \"global-pull-secret-syncer-z9ws7\" (UID: \"8c764213-65e7-4296-a8b6-023c581b36c1\") " pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:38.020053 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:38.020002    2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jun 15 05:48:38.020099 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:38.020056    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c764213-65e7-4296-a8b6-023c581b36c1-original-pull-secret podName:8c764213-65e7-4296-a8b6-023c581b36c1 nodeName:}" failed. No retries permitted until 2026-06-15 05:49:10.020041739 +0000 UTC m=+65.168221719 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8c764213-65e7-4296-a8b6-023c581b36c1-original-pull-secret") pod "global-pull-secret-syncer-z9ws7" (UID: "8c764213-65e7-4296-a8b6-023c581b36c1") : object "kube-system"/"original-pull-secret" not registered
Jun 15 05:48:38.120613 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:38.120569    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57565893-704b-4974-bc5b-c8f9cb932d15-metrics-certs\") pod \"network-metrics-daemon-vs8ck\" (UID: \"57565893-704b-4974-bc5b-c8f9cb932d15\") " pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:38.120790 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:38.120732    2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:48:38.120855 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:38.120820    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57565893-704b-4974-bc5b-c8f9cb932d15-metrics-certs podName:57565893-704b-4974-bc5b-c8f9cb932d15 nodeName:}" failed. No retries permitted until 2026-06-15 05:49:10.120800671 +0000 UTC m=+65.268980646 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/57565893-704b-4974-bc5b-c8f9cb932d15-metrics-certs") pod "network-metrics-daemon-vs8ck" (UID: "57565893-704b-4974-bc5b-c8f9cb932d15") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jun 15 05:48:38.221845 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:38.221810    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mws52\" (UniqueName: \"kubernetes.io/projected/243c1d92-89f7-4ebb-838a-0f18bd3216d8-kube-api-access-mws52\") pod \"network-check-target-9b7jj\" (UID: \"243c1d92-89f7-4ebb-838a-0f18bd3216d8\") " pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:38.222021 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:38.221955    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jun 15 05:48:38.222021 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:38.221971    2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jun 15 05:48:38.222021 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:38.221980    2572 projected.go:196] Error preparing data for projected volume kube-api-access-mws52 for pod openshift-network-diagnostics/network-check-target-9b7jj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:48:38.222162 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:38.222026    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/243c1d92-89f7-4ebb-838a-0f18bd3216d8-kube-api-access-mws52 podName:243c1d92-89f7-4ebb-838a-0f18bd3216d8 nodeName:}" failed. No retries permitted until 2026-06-15 05:49:10.222012835 +0000 UTC m=+65.370192810 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mws52" (UniqueName: "kubernetes.io/projected/243c1d92-89f7-4ebb-838a-0f18bd3216d8-kube-api-access-mws52") pod "network-check-target-9b7jj" (UID: "243c1d92-89f7-4ebb-838a-0f18bd3216d8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jun 15 05:48:38.318592 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:38.318510    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:38.318592 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:38.318550    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:38.318801 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:38.318510    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:38.318801 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:38.318634    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9ws7" podUID="8c764213-65e7-4296-a8b6-023c581b36c1"
Jun 15 05:48:38.318801 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:38.318709    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b7jj" podUID="243c1d92-89f7-4ebb-838a-0f18bd3216d8"
Jun 15 05:48:38.318801 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:38.318789    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vs8ck" podUID="57565893-704b-4974-bc5b-c8f9cb932d15"
Jun 15 05:48:40.318446 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:40.318409    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:40.318947 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:40.318417    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:40.318947 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:40.318409    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:40.318947 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:40.318565    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vs8ck" podUID="57565893-704b-4974-bc5b-c8f9cb932d15"
Jun 15 05:48:40.318947 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:40.318670    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b7jj" podUID="243c1d92-89f7-4ebb-838a-0f18bd3216d8"
Jun 15 05:48:40.318947 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:40.318746    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9ws7" podUID="8c764213-65e7-4296-a8b6-023c581b36c1"
Jun 15 05:48:41.637129 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:41.637101    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jwgfl" event={"ID":"634277cb-f6c8-4254-b1aa-ccaebfe829a9","Type":"ContainerStarted","Data":"10d30711e30ff919c494154a8e2038c24531a3cacd846b8bf957042c1775beaa"}
Jun 15 05:48:42.319028 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:42.318995    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:42.319201 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:42.319095    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:42.319201 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:42.319188    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vs8ck" podUID="57565893-704b-4974-bc5b-c8f9cb932d15"
Jun 15 05:48:42.319338 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:42.319094    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b7jj" podUID="243c1d92-89f7-4ebb-838a-0f18bd3216d8"
Jun 15 05:48:42.319338 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:42.319091    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:42.319338 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:42.319272    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9ws7" podUID="8c764213-65e7-4296-a8b6-023c581b36c1"
Jun 15 05:48:42.642046 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:42.642010    2572 generic.go:356] "Generic (PLEG): container finished" podID="634277cb-f6c8-4254-b1aa-ccaebfe829a9" containerID="10d30711e30ff919c494154a8e2038c24531a3cacd846b8bf957042c1775beaa" exitCode=0
Jun 15 05:48:42.642566 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:42.642078    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jwgfl" event={"ID":"634277cb-f6c8-4254-b1aa-ccaebfe829a9","Type":"ContainerDied","Data":"10d30711e30ff919c494154a8e2038c24531a3cacd846b8bf957042c1775beaa"}
Jun 15 05:48:43.646985 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:43.646955    2572 generic.go:356] "Generic (PLEG): container finished" podID="634277cb-f6c8-4254-b1aa-ccaebfe829a9" containerID="3325d69e6d0877afd6a53dd887cecc71f33e5302d206778a4b7cbacab303d8d0" exitCode=0
Jun 15 05:48:43.647546 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:43.647014    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jwgfl" event={"ID":"634277cb-f6c8-4254-b1aa-ccaebfe829a9","Type":"ContainerDied","Data":"3325d69e6d0877afd6a53dd887cecc71f33e5302d206778a4b7cbacab303d8d0"}
Jun 15 05:48:44.318607 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:44.318573    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:44.318803 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:44.318572    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:44.318803 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:44.318690    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b7jj" podUID="243c1d92-89f7-4ebb-838a-0f18bd3216d8"
Jun 15 05:48:44.318803 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:44.318572    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:44.318803 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:44.318756    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vs8ck" podUID="57565893-704b-4974-bc5b-c8f9cb932d15"
Jun 15 05:48:44.319014 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:44.318814    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9ws7" podUID="8c764213-65e7-4296-a8b6-023c581b36c1"
Jun 15 05:48:44.652026 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:44.651993    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jwgfl" event={"ID":"634277cb-f6c8-4254-b1aa-ccaebfe829a9","Type":"ContainerStarted","Data":"763d1fd3e7112ef599dabfb86519e98a849f6503b5f674ef5200740a0f69f449"}
Jun 15 05:48:44.680846 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:44.680803    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jwgfl" podStartSLOduration=9.767491967 podStartE2EDuration="39.680786888s" totalImagesPullingTime="29.913294921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:48:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:48:44.680605853 +0000 UTC m=+39.828785851" watchObservedRunningTime="2026-06-15 05:48:44.680786888 +0000 UTC m=+39.828966885"
Jun 15 05:48:46.318624 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:46.318593    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:46.319277 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:46.318643    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:46.319277 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:46.318603    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:46.319277 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:46.318708    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b7jj" podUID="243c1d92-89f7-4ebb-838a-0f18bd3216d8"
Jun 15 05:48:46.319277 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:46.318792    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vs8ck" podUID="57565893-704b-4974-bc5b-c8f9cb932d15"
Jun 15 05:48:46.319277 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:46.318926    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9ws7" podUID="8c764213-65e7-4296-a8b6-023c581b36c1"
Jun 15 05:48:48.318840 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:48.318800    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:48.318840 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:48.318824    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:48.319335 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:48.318812    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:48.319335 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:48.318921    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b7jj" podUID="243c1d92-89f7-4ebb-838a-0f18bd3216d8"
Jun 15 05:48:48.319335 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:48.318973    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vs8ck" podUID="57565893-704b-4974-bc5b-c8f9cb932d15"
Jun 15 05:48:48.319335 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:48.319029    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9ws7" podUID="8c764213-65e7-4296-a8b6-023c581b36c1"
Jun 15 05:48:50.318406 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:50.318380    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:50.318852 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:50.318474    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:50.318852 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:50.318484    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b7jj" podUID="243c1d92-89f7-4ebb-838a-0f18bd3216d8"
Jun 15 05:48:50.318852 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:50.318518    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:50.318852 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:50.318622    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9ws7" podUID="8c764213-65e7-4296-a8b6-023c581b36c1"
Jun 15 05:48:50.318852 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:50.318757    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vs8ck" podUID="57565893-704b-4974-bc5b-c8f9cb932d15"
Jun 15 05:48:50.798630 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:50.798586    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z9ws7"]
Jun 15 05:48:50.798816 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:50.798701    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:50.798816 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:50.798788    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9ws7" podUID="8c764213-65e7-4296-a8b6-023c581b36c1"
Jun 15 05:48:50.799260 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:50.799237    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vs8ck"]
Jun 15 05:48:50.799342 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:50.799334    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:50.799476 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:50.799455    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vs8ck" podUID="57565893-704b-4974-bc5b-c8f9cb932d15"
Jun 15 05:48:50.799888 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:50.799857    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9b7jj"]
Jun 15 05:48:50.799998 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:50.799954    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:50.800050 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:50.800034    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b7jj" podUID="243c1d92-89f7-4ebb-838a-0f18bd3216d8"
Jun 15 05:48:52.318997 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:52.318963    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:52.319574 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:52.319055    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:52.319574 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:52.319062    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b7jj" podUID="243c1d92-89f7-4ebb-838a-0f18bd3216d8"
Jun 15 05:48:52.319574 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:52.319084    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:52.319574 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:52.319145    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9ws7" podUID="8c764213-65e7-4296-a8b6-023c581b36c1"
Jun 15 05:48:52.319574 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:52.319205    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vs8ck" podUID="57565893-704b-4974-bc5b-c8f9cb932d15"
Jun 15 05:48:54.318830 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:54.318635    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:54.319210 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:54.318656    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:54.319210 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:54.318906    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-9b7jj" podUID="243c1d92-89f7-4ebb-838a-0f18bd3216d8"
Jun 15 05:48:54.319210 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:54.318700    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:54.319210 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:54.318983    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-z9ws7" podUID="8c764213-65e7-4296-a8b6-023c581b36c1"
Jun 15 05:48:54.319210 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:54.319068    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vs8ck" podUID="57565893-704b-4974-bc5b-c8f9cb932d15"
Jun 15 05:48:55.150923 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.150894    2572 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-16.ec2.internal" event="NodeReady"
Jun 15 05:48:55.151107 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.151028    2572 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jun 15 05:48:55.204762 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.204735    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-q6c87"]
Jun 15 05:48:55.224313 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.224246    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-k5l48"]
Jun 15 05:48:55.224461 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.224382    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q6c87"
Jun 15 05:48:55.226607 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.226585    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bq4gf\""
Jun 15 05:48:55.226724 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.226592    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jun 15 05:48:55.226724 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.226592    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jun 15 05:48:55.250343 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.250315    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q6c87"]
Jun 15 05:48:55.250498 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.250350    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k5l48"]
Jun 15 05:48:55.250498 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.250451    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k5l48"
Jun 15 05:48:55.250600 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.250508    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47e37b20-ad06-4ff6-9beb-6503464f6ef8-metrics-tls\") pod \"dns-default-q6c87\" (UID: \"47e37b20-ad06-4ff6-9beb-6503464f6ef8\") " pod="openshift-dns/dns-default-q6c87"
Jun 15 05:48:55.250600 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.250551    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/47e37b20-ad06-4ff6-9beb-6503464f6ef8-config-volume\") pod \"dns-default-q6c87\" (UID: \"47e37b20-ad06-4ff6-9beb-6503464f6ef8\") " pod="openshift-dns/dns-default-q6c87"
Jun 15 05:48:55.250702 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.250630    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd7pz\" (UniqueName: \"kubernetes.io/projected/47e37b20-ad06-4ff6-9beb-6503464f6ef8-kube-api-access-cd7pz\") pod \"dns-default-q6c87\" (UID: \"47e37b20-ad06-4ff6-9beb-6503464f6ef8\") " pod="openshift-dns/dns-default-q6c87"
Jun 15 05:48:55.250702 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.250688    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47e37b20-ad06-4ff6-9beb-6503464f6ef8-tmp-dir\") pod \"dns-default-q6c87\" (UID: \"47e37b20-ad06-4ff6-9beb-6503464f6ef8\") " pod="openshift-dns/dns-default-q6c87"
Jun 15 05:48:55.252751 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.252731    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jun 15 05:48:55.252855 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.252821    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jun 15 05:48:55.252855 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.252834    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jun 15 05:48:55.252855 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.252835    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-z7kzt\""
Jun 15 05:48:55.306576 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.306547    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-xpdg2"]
Jun 15 05:48:55.327864 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.327839    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xpdg2"]
Jun 15 05:48:55.328215 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.327979    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xpdg2"
Jun 15 05:48:55.330139 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.330117    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jun 15 05:48:55.330302 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.330277    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fsjw7\""
Jun 15 05:48:55.330444 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.330393    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jun 15 05:48:55.330444 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.330406    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jun 15 05:48:55.330560 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.330451    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jun 15 05:48:55.351722 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.351698    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xnwv\" (UniqueName: \"kubernetes.io/projected/bdbef6be-323c-4a82-9f0a-8bb8fdafc620-kube-api-access-4xnwv\") pod \"insights-runtime-extractor-xpdg2\" (UID: \"bdbef6be-323c-4a82-9f0a-8bb8fdafc620\") " pod="openshift-insights/insights-runtime-extractor-xpdg2"
Jun 15 05:48:55.351852 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.351730    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd7pz\" (UniqueName: \"kubernetes.io/projected/47e37b20-ad06-4ff6-9beb-6503464f6ef8-kube-api-access-cd7pz\") pod \"dns-default-q6c87\" (UID: \"47e37b20-ad06-4ff6-9beb-6503464f6ef8\") " pod="openshift-dns/dns-default-q6c87"
Jun 15 05:48:55.351852 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.351748    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcbzq\" (UniqueName: \"kubernetes.io/projected/6f7e3592-7192-4acb-a079-7cc24646f8a5-kube-api-access-dcbzq\") pod \"ingress-canary-k5l48\" (UID: \"6f7e3592-7192-4acb-a079-7cc24646f8a5\") " pod="openshift-ingress-canary/ingress-canary-k5l48"
Jun 15 05:48:55.351852 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.351775    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47e37b20-ad06-4ff6-9beb-6503464f6ef8-tmp-dir\") pod \"dns-default-q6c87\" (UID: \"47e37b20-ad06-4ff6-9beb-6503464f6ef8\") " pod="openshift-dns/dns-default-q6c87"
Jun 15 05:48:55.351852 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.351792    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bdbef6be-323c-4a82-9f0a-8bb8fdafc620-crio-socket\") pod \"insights-runtime-extractor-xpdg2\" (UID: \"bdbef6be-323c-4a82-9f0a-8bb8fdafc620\") " pod="openshift-insights/insights-runtime-extractor-xpdg2"
Jun 15 05:48:55.351852 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.351808    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bdbef6be-323c-4a82-9f0a-8bb8fdafc620-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xpdg2\" (UID: \"bdbef6be-323c-4a82-9f0a-8bb8fdafc620\") " pod="openshift-insights/insights-runtime-extractor-xpdg2"
Jun 15 05:48:55.351852 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.351836    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f7e3592-7192-4acb-a079-7cc24646f8a5-cert\") pod \"ingress-canary-k5l48\" (UID: \"6f7e3592-7192-4acb-a079-7cc24646f8a5\") " pod="openshift-ingress-canary/ingress-canary-k5l48"
Jun 15 05:48:55.351852 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.351853    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47e37b20-ad06-4ff6-9beb-6503464f6ef8-metrics-tls\") pod \"dns-default-q6c87\" (UID: \"47e37b20-ad06-4ff6-9beb-6503464f6ef8\") " pod="openshift-dns/dns-default-q6c87"
Jun 15 05:48:55.352173 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.351933    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/47e37b20-ad06-4ff6-9beb-6503464f6ef8-config-volume\") pod \"dns-default-q6c87\" (UID: \"47e37b20-ad06-4ff6-9beb-6503464f6ef8\") " pod="openshift-dns/dns-default-q6c87"
Jun 15 05:48:55.352173 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.351991    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bdbef6be-323c-4a82-9f0a-8bb8fdafc620-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xpdg2\" (UID: \"bdbef6be-323c-4a82-9f0a-8bb8fdafc620\") " pod="openshift-insights/insights-runtime-extractor-xpdg2"
Jun 15 05:48:55.352173 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.352079    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/47e37b20-ad06-4ff6-9beb-6503464f6ef8-tmp-dir\") pod \"dns-default-q6c87\" (UID: \"47e37b20-ad06-4ff6-9beb-6503464f6ef8\") " pod="openshift-dns/dns-default-q6c87"
Jun 15 05:48:55.352173 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.352121    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bdbef6be-323c-4a82-9f0a-8bb8fdafc620-data-volume\") pod \"insights-runtime-extractor-xpdg2\" (UID: \"bdbef6be-323c-4a82-9f0a-8bb8fdafc620\") " pod="openshift-insights/insights-runtime-extractor-xpdg2"
Jun 15 05:48:55.352503 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.352481    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/47e37b20-ad06-4ff6-9beb-6503464f6ef8-config-volume\") pod \"dns-default-q6c87\" (UID: \"47e37b20-ad06-4ff6-9beb-6503464f6ef8\") " pod="openshift-dns/dns-default-q6c87"
Jun 15 05:48:55.355748 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.355732    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/47e37b20-ad06-4ff6-9beb-6503464f6ef8-metrics-tls\") pod \"dns-default-q6c87\" (UID: \"47e37b20-ad06-4ff6-9beb-6503464f6ef8\") " pod="openshift-dns/dns-default-q6c87"
Jun 15 05:48:55.360949 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.360929    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd7pz\" (UniqueName: \"kubernetes.io/projected/47e37b20-ad06-4ff6-9beb-6503464f6ef8-kube-api-access-cd7pz\") pod \"dns-default-q6c87\" (UID: \"47e37b20-ad06-4ff6-9beb-6503464f6ef8\") " pod="openshift-dns/dns-default-q6c87"
Jun 15 05:48:55.452953 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.452916    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f7e3592-7192-4acb-a079-7cc24646f8a5-cert\") pod \"ingress-canary-k5l48\" (UID: \"6f7e3592-7192-4acb-a079-7cc24646f8a5\") " pod="openshift-ingress-canary/ingress-canary-k5l48"
Jun 15 05:48:55.453146 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.452969    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bdbef6be-323c-4a82-9f0a-8bb8fdafc620-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xpdg2\" (UID: \"bdbef6be-323c-4a82-9f0a-8bb8fdafc620\") " pod="openshift-insights/insights-runtime-extractor-xpdg2"
Jun 15 05:48:55.453146 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.453030    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bdbef6be-323c-4a82-9f0a-8bb8fdafc620-data-volume\") pod \"insights-runtime-extractor-xpdg2\" (UID: \"bdbef6be-323c-4a82-9f0a-8bb8fdafc620\") " pod="openshift-insights/insights-runtime-extractor-xpdg2"
Jun 15 05:48:55.453146 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.453082    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xnwv\" (UniqueName: \"kubernetes.io/projected/bdbef6be-323c-4a82-9f0a-8bb8fdafc620-kube-api-access-4xnwv\") pod \"insights-runtime-extractor-xpdg2\" (UID: \"bdbef6be-323c-4a82-9f0a-8bb8fdafc620\") " pod="openshift-insights/insights-runtime-extractor-xpdg2"
Jun 15 05:48:55.453146 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.453110    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcbzq\" (UniqueName: \"kubernetes.io/projected/6f7e3592-7192-4acb-a079-7cc24646f8a5-kube-api-access-dcbzq\") pod \"ingress-canary-k5l48\" (UID: \"6f7e3592-7192-4acb-a079-7cc24646f8a5\") " pod="openshift-ingress-canary/ingress-canary-k5l48"
Jun 15 05:48:55.453333 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.453146    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bdbef6be-323c-4a82-9f0a-8bb8fdafc620-crio-socket\") pod \"insights-runtime-extractor-xpdg2\" (UID: \"bdbef6be-323c-4a82-9f0a-8bb8fdafc620\") " pod="openshift-insights/insights-runtime-extractor-xpdg2"
Jun 15 05:48:55.453333 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.453173    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bdbef6be-323c-4a82-9f0a-8bb8fdafc620-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xpdg2\" (UID: \"bdbef6be-323c-4a82-9f0a-8bb8fdafc620\") " pod="openshift-insights/insights-runtime-extractor-xpdg2"
Jun 15 05:48:55.453457 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.453411    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bdbef6be-323c-4a82-9f0a-8bb8fdafc620-crio-socket\") pod \"insights-runtime-extractor-xpdg2\" (UID: \"bdbef6be-323c-4a82-9f0a-8bb8fdafc620\") " pod="openshift-insights/insights-runtime-extractor-xpdg2"
Jun 15 05:48:55.453457 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.453409    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bdbef6be-323c-4a82-9f0a-8bb8fdafc620-data-volume\") pod \"insights-runtime-extractor-xpdg2\" (UID: \"bdbef6be-323c-4a82-9f0a-8bb8fdafc620\") " pod="openshift-insights/insights-runtime-extractor-xpdg2"
Jun 15 05:48:55.453574 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.453554    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bdbef6be-323c-4a82-9f0a-8bb8fdafc620-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-xpdg2\" (UID: \"bdbef6be-323c-4a82-9f0a-8bb8fdafc620\") " pod="openshift-insights/insights-runtime-extractor-xpdg2"
Jun 15 05:48:55.455575 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.455555    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bdbef6be-323c-4a82-9f0a-8bb8fdafc620-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-xpdg2\" (UID: \"bdbef6be-323c-4a82-9f0a-8bb8fdafc620\") " pod="openshift-insights/insights-runtime-extractor-xpdg2"
Jun 15 05:48:55.455725 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.455705    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6f7e3592-7192-4acb-a079-7cc24646f8a5-cert\") pod \"ingress-canary-k5l48\" (UID: \"6f7e3592-7192-4acb-a079-7cc24646f8a5\") " pod="openshift-ingress-canary/ingress-canary-k5l48"
Jun 15 05:48:55.465475 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.465457    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcbzq\" (UniqueName: \"kubernetes.io/projected/6f7e3592-7192-4acb-a079-7cc24646f8a5-kube-api-access-dcbzq\") pod \"ingress-canary-k5l48\" (UID: \"6f7e3592-7192-4acb-a079-7cc24646f8a5\") " pod="openshift-ingress-canary/ingress-canary-k5l48"
Jun 15 05:48:55.466404 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.466385    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xnwv\" (UniqueName: \"kubernetes.io/projected/bdbef6be-323c-4a82-9f0a-8bb8fdafc620-kube-api-access-4xnwv\") pod \"insights-runtime-extractor-xpdg2\" (UID: \"bdbef6be-323c-4a82-9f0a-8bb8fdafc620\") " pod="openshift-insights/insights-runtime-extractor-xpdg2"
Jun 15 05:48:55.535388 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.535281    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q6c87"
Jun 15 05:48:55.560754 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.560724    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-k5l48"
Jun 15 05:48:55.636348 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.636311    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-xpdg2"
Jun 15 05:48:55.771583 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.771555    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q6c87"]
Jun 15 05:48:55.775737 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.775715    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-k5l48"]
Jun 15 05:48:55.778524 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:55.778484    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f7e3592_7192_4acb_a079_7cc24646f8a5.slice/crio-c21ca7e3d02b7dacc4410483957f349174ac8a9b8036a59905cd0a868bfb0620 WatchSource:0}: Error finding container c21ca7e3d02b7dacc4410483957f349174ac8a9b8036a59905cd0a868bfb0620: Status 404 returned error can't find the container with id c21ca7e3d02b7dacc4410483957f349174ac8a9b8036a59905cd0a868bfb0620
Jun 15 05:48:55.813862 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:55.813793    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-xpdg2"]
Jun 15 05:48:55.817083 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:48:55.817048    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdbef6be_323c_4a82_9f0a_8bb8fdafc620.slice/crio-95b2e40857cb853ee6568a5c6d4967fc1cc5cd4b20e948527370c18e2a1d0b8d WatchSource:0}: Error finding container 95b2e40857cb853ee6568a5c6d4967fc1cc5cd4b20e948527370c18e2a1d0b8d: Status 404 returned error can't find the container with id 95b2e40857cb853ee6568a5c6d4967fc1cc5cd4b20e948527370c18e2a1d0b8d
Jun 15 05:48:56.319162 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:56.319129    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:48:56.319352 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:56.319129    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:48:56.319598 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:56.319149    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:48:56.321988 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:56.321964    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jun 15 05:48:56.322639 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:56.322594    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jun 15 05:48:56.323111 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:56.322911    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jun 15 05:48:56.323111 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:56.322983    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jun 15 05:48:56.323111 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:56.323005    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sh6m9\""
Jun 15 05:48:56.323380 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:56.323322    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4mr6w\""
Jun 15 05:48:56.684952 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:56.683212    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k5l48" event={"ID":"6f7e3592-7192-4acb-a079-7cc24646f8a5","Type":"ContainerStarted","Data":"c21ca7e3d02b7dacc4410483957f349174ac8a9b8036a59905cd0a868bfb0620"}
Jun 15 05:48:56.684952 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:56.684582    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xpdg2" event={"ID":"bdbef6be-323c-4a82-9f0a-8bb8fdafc620","Type":"ContainerStarted","Data":"95b2e40857cb853ee6568a5c6d4967fc1cc5cd4b20e948527370c18e2a1d0b8d"}
Jun 15 05:48:56.686209 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:56.686179    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q6c87" event={"ID":"47e37b20-ad06-4ff6-9beb-6503464f6ef8","Type":"ContainerStarted","Data":"2c841bfd3684855a9da89aa2e5608b7e4826d647cf1f2db1ee03457c61ccb829"}
Jun 15 05:48:57.096275 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:57.096199    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-864fcfc6f9-54pzw"]
Jun 15 05:48:57.119044 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:57.119008    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-864fcfc6f9-54pzw"]
Jun 15 05:48:57.119204 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:57.119162    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-864fcfc6f9-54pzw"
Jun 15 05:48:57.122097 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:57.121886    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jun 15 05:48:57.122097 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:57.121922    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jun 15 05:48:57.122097 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:57.121963    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-62krm\""
Jun 15 05:48:57.122346 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:57.122154    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jun 15 05:48:57.122346 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:57.122182    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jun 15 05:48:57.123040 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:57.123021    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jun 15 05:48:57.167212 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:57.167181    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96f776da-6137-4f8b-8324-e6cab29e36d6-metrics-client-ca\") pod \"prometheus-operator-864fcfc6f9-54pzw\" (UID: \"96f776da-6137-4f8b-8324-e6cab29e36d6\") " pod="openshift-monitoring/prometheus-operator-864fcfc6f9-54pzw"
Jun 15 05:48:57.167450 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:57.167250    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/96f776da-6137-4f8b-8324-e6cab29e36d6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-864fcfc6f9-54pzw\" (UID: \"96f776da-6137-4f8b-8324-e6cab29e36d6\") " pod="openshift-monitoring/prometheus-operator-864fcfc6f9-54pzw"
Jun 15 05:48:57.167450 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:57.167319    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/96f776da-6137-4f8b-8324-e6cab29e36d6-prometheus-operator-tls\") pod \"prometheus-operator-864fcfc6f9-54pzw\" (UID: \"96f776da-6137-4f8b-8324-e6cab29e36d6\") " pod="openshift-monitoring/prometheus-operator-864fcfc6f9-54pzw"
Jun 15 05:48:57.167450 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:57.167382    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtvzw\" (UniqueName: \"kubernetes.io/projected/96f776da-6137-4f8b-8324-e6cab29e36d6-kube-api-access-qtvzw\") pod \"prometheus-operator-864fcfc6f9-54pzw\" (UID: \"96f776da-6137-4f8b-8324-e6cab29e36d6\") " pod="openshift-monitoring/prometheus-operator-864fcfc6f9-54pzw"
Jun 15 05:48:57.268851 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:57.268325    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/96f776da-6137-4f8b-8324-e6cab29e36d6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-864fcfc6f9-54pzw\" (UID: \"96f776da-6137-4f8b-8324-e6cab29e36d6\") " pod="openshift-monitoring/prometheus-operator-864fcfc6f9-54pzw"
Jun 15 05:48:57.268851 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:57.268622    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/96f776da-6137-4f8b-8324-e6cab29e36d6-prometheus-operator-tls\") pod \"prometheus-operator-864fcfc6f9-54pzw\" (UID: \"96f776da-6137-4f8b-8324-e6cab29e36d6\") " pod="openshift-monitoring/prometheus-operator-864fcfc6f9-54pzw"
Jun 15 05:48:57.268851 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:57.268672    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtvzw\" (UniqueName: \"kubernetes.io/projected/96f776da-6137-4f8b-8324-e6cab29e36d6-kube-api-access-qtvzw\") pod \"prometheus-operator-864fcfc6f9-54pzw\" (UID: \"96f776da-6137-4f8b-8324-e6cab29e36d6\") " pod="openshift-monitoring/prometheus-operator-864fcfc6f9-54pzw"
Jun 15 05:48:57.268851 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:57.268732    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96f776da-6137-4f8b-8324-e6cab29e36d6-metrics-client-ca\") pod \"prometheus-operator-864fcfc6f9-54pzw\" (UID: \"96f776da-6137-4f8b-8324-e6cab29e36d6\") " pod="openshift-monitoring/prometheus-operator-864fcfc6f9-54pzw"
Jun 15 05:48:57.268851 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:57.268776    2572 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jun 15 05:48:57.268851 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:48:57.268850    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96f776da-6137-4f8b-8324-e6cab29e36d6-prometheus-operator-tls podName:96f776da-6137-4f8b-8324-e6cab29e36d6 nodeName:}" failed. No retries permitted until 2026-06-15 05:48:57.76882923 +0000 UTC m=+52.917009212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/96f776da-6137-4f8b-8324-e6cab29e36d6-prometheus-operator-tls") pod "prometheus-operator-864fcfc6f9-54pzw" (UID: "96f776da-6137-4f8b-8324-e6cab29e36d6") : secret "prometheus-operator-tls" not found
Jun 15 05:48:57.269934 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:57.269890    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/96f776da-6137-4f8b-8324-e6cab29e36d6-metrics-client-ca\") pod \"prometheus-operator-864fcfc6f9-54pzw\" (UID: \"96f776da-6137-4f8b-8324-e6cab29e36d6\") " pod="openshift-monitoring/prometheus-operator-864fcfc6f9-54pzw"
Jun 15 05:48:57.272842 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:57.272822    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/96f776da-6137-4f8b-8324-e6cab29e36d6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-864fcfc6f9-54pzw\" (UID: \"96f776da-6137-4f8b-8324-e6cab29e36d6\") " pod="openshift-monitoring/prometheus-operator-864fcfc6f9-54pzw"
Jun 15 05:48:57.281023 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:57.280999    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtvzw\" (UniqueName: \"kubernetes.io/projected/96f776da-6137-4f8b-8324-e6cab29e36d6-kube-api-access-qtvzw\") pod \"prometheus-operator-864fcfc6f9-54pzw\" (UID: \"96f776da-6137-4f8b-8324-e6cab29e36d6\") " pod="openshift-monitoring/prometheus-operator-864fcfc6f9-54pzw"
Jun 15 05:48:57.772567 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:57.772535    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/96f776da-6137-4f8b-8324-e6cab29e36d6-prometheus-operator-tls\") pod \"prometheus-operator-864fcfc6f9-54pzw\" (UID: \"96f776da-6137-4f8b-8324-e6cab29e36d6\") " pod="openshift-monitoring/prometheus-operator-864fcfc6f9-54pzw"
Jun 15 05:48:57.775219 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:57.775184    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/96f776da-6137-4f8b-8324-e6cab29e36d6-prometheus-operator-tls\") pod \"prometheus-operator-864fcfc6f9-54pzw\" (UID: \"96f776da-6137-4f8b-8324-e6cab29e36d6\") " pod="openshift-monitoring/prometheus-operator-864fcfc6f9-54pzw"
Jun 15 05:48:58.031908 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:58.031832    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-864fcfc6f9-54pzw"
Jun 15 05:48:59.639283 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.639247    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65b6bfd656-g7zrp"]
Jun 15 05:48:59.802229 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.802196    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65b6bfd656-g7zrp"]
Jun 15 05:48:59.802413 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.802296    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65b6bfd656-g7zrp"
Jun 15 05:48:59.804687 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.804653    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jun 15 05:48:59.804847 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.804740    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jun 15 05:48:59.804847 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.804784    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qvwbv\""
Jun 15 05:48:59.804847 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.804787    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jun 15 05:48:59.805312 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.805279    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jun 15 05:48:59.805312 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.805308    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jun 15 05:48:59.805660 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.805324    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jun 15 05:48:59.805660 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.805351    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jun 15 05:48:59.810332 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.810313    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jun 15 05:48:59.890441 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.890413    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77cb750c-2c2f-41b9-ada6-7021f53ea271-trusted-ca-bundle\") pod \"console-65b6bfd656-g7zrp\" (UID: \"77cb750c-2c2f-41b9-ada6-7021f53ea271\") " pod="openshift-console/console-65b6bfd656-g7zrp"
Jun 15 05:48:59.890578 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.890450    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/77cb750c-2c2f-41b9-ada6-7021f53ea271-oauth-serving-cert\") pod \"console-65b6bfd656-g7zrp\" (UID: \"77cb750c-2c2f-41b9-ada6-7021f53ea271\") " pod="openshift-console/console-65b6bfd656-g7zrp"
Jun 15 05:48:59.890578 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.890476    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx8jw\" (UniqueName: \"kubernetes.io/projected/77cb750c-2c2f-41b9-ada6-7021f53ea271-kube-api-access-vx8jw\") pod \"console-65b6bfd656-g7zrp\" (UID: \"77cb750c-2c2f-41b9-ada6-7021f53ea271\") " pod="openshift-console/console-65b6bfd656-g7zrp"
Jun 15 05:48:59.890658 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.890605    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/77cb750c-2c2f-41b9-ada6-7021f53ea271-console-config\") pod \"console-65b6bfd656-g7zrp\" (UID: \"77cb750c-2c2f-41b9-ada6-7021f53ea271\") " pod="openshift-console/console-65b6bfd656-g7zrp"
Jun 15 05:48:59.890658 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.890633    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/77cb750c-2c2f-41b9-ada6-7021f53ea271-service-ca\") pod \"console-65b6bfd656-g7zrp\" (UID: \"77cb750c-2c2f-41b9-ada6-7021f53ea271\") " pod="openshift-console/console-65b6bfd656-g7zrp"
Jun 15 05:48:59.890721 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.890666    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/77cb750c-2c2f-41b9-ada6-7021f53ea271-console-serving-cert\") pod \"console-65b6bfd656-g7zrp\" (UID: \"77cb750c-2c2f-41b9-ada6-7021f53ea271\") " pod="openshift-console/console-65b6bfd656-g7zrp"
Jun 15 05:48:59.890752 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.890735    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/77cb750c-2c2f-41b9-ada6-7021f53ea271-console-oauth-config\") pod \"console-65b6bfd656-g7zrp\" (UID: \"77cb750c-2c2f-41b9-ada6-7021f53ea271\") " pod="openshift-console/console-65b6bfd656-g7zrp"
Jun 15 05:48:59.991324 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.991292    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/77cb750c-2c2f-41b9-ada6-7021f53ea271-console-config\") pod \"console-65b6bfd656-g7zrp\" (UID: \"77cb750c-2c2f-41b9-ada6-7021f53ea271\") " pod="openshift-console/console-65b6bfd656-g7zrp"
Jun 15 05:48:59.991324 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.991322    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/77cb750c-2c2f-41b9-ada6-7021f53ea271-service-ca\") pod \"console-65b6bfd656-g7zrp\" (UID: \"77cb750c-2c2f-41b9-ada6-7021f53ea271\") " pod="openshift-console/console-65b6bfd656-g7zrp"
Jun 15 05:48:59.991578 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.991343    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/77cb750c-2c2f-41b9-ada6-7021f53ea271-console-serving-cert\") pod \"console-65b6bfd656-g7zrp\" (UID: \"77cb750c-2c2f-41b9-ada6-7021f53ea271\") " pod="openshift-console/console-65b6bfd656-g7zrp"
Jun 15 05:48:59.991578 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.991401    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/77cb750c-2c2f-41b9-ada6-7021f53ea271-console-oauth-config\") pod \"console-65b6bfd656-g7zrp\" (UID: \"77cb750c-2c2f-41b9-ada6-7021f53ea271\") " pod="openshift-console/console-65b6bfd656-g7zrp"
Jun 15 05:48:59.991578 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.991421    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77cb750c-2c2f-41b9-ada6-7021f53ea271-trusted-ca-bundle\") pod \"console-65b6bfd656-g7zrp\" (UID: \"77cb750c-2c2f-41b9-ada6-7021f53ea271\") " pod="openshift-console/console-65b6bfd656-g7zrp"
Jun 15 05:48:59.991578 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.991446    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/77cb750c-2c2f-41b9-ada6-7021f53ea271-oauth-serving-cert\") pod \"console-65b6bfd656-g7zrp\" (UID: \"77cb750c-2c2f-41b9-ada6-7021f53ea271\") " pod="openshift-console/console-65b6bfd656-g7zrp"
Jun 15 05:48:59.991801 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.991640    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx8jw\" (UniqueName: \"kubernetes.io/projected/77cb750c-2c2f-41b9-ada6-7021f53ea271-kube-api-access-vx8jw\") pod \"console-65b6bfd656-g7zrp\" (UID: \"77cb750c-2c2f-41b9-ada6-7021f53ea271\") " pod="openshift-console/console-65b6bfd656-g7zrp"
Jun 15 05:48:59.992116 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.992094    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/77cb750c-2c2f-41b9-ada6-7021f53ea271-service-ca\") pod \"console-65b6bfd656-g7zrp\" (UID: \"77cb750c-2c2f-41b9-ada6-7021f53ea271\") " pod="openshift-console/console-65b6bfd656-g7zrp"
Jun 15 05:48:59.992216 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.992183    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/77cb750c-2c2f-41b9-ada6-7021f53ea271-console-config\") pod \"console-65b6bfd656-g7zrp\" (UID: \"77cb750c-2c2f-41b9-ada6-7021f53ea271\") " pod="openshift-console/console-65b6bfd656-g7zrp"
Jun 15 05:48:59.992216 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.992190    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/77cb750c-2c2f-41b9-ada6-7021f53ea271-oauth-serving-cert\") pod \"console-65b6bfd656-g7zrp\" (UID: \"77cb750c-2c2f-41b9-ada6-7021f53ea271\") " pod="openshift-console/console-65b6bfd656-g7zrp"
Jun 15 05:48:59.992379 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.992344    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77cb750c-2c2f-41b9-ada6-7021f53ea271-trusted-ca-bundle\") pod \"console-65b6bfd656-g7zrp\" (UID: \"77cb750c-2c2f-41b9-ada6-7021f53ea271\") " pod="openshift-console/console-65b6bfd656-g7zrp"
Jun 15 05:48:59.994010 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.993986    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/77cb750c-2c2f-41b9-ada6-7021f53ea271-console-oauth-config\") pod \"console-65b6bfd656-g7zrp\" (UID: \"77cb750c-2c2f-41b9-ada6-7021f53ea271\") " pod="openshift-console/console-65b6bfd656-g7zrp"
Jun 15 05:48:59.994089 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:48:59.994070    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/77cb750c-2c2f-41b9-ada6-7021f53ea271-console-serving-cert\") pod \"console-65b6bfd656-g7zrp\" (UID: \"77cb750c-2c2f-41b9-ada6-7021f53ea271\") " pod="openshift-console/console-65b6bfd656-g7zrp"
Jun 15 05:49:00.000766 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:00.000746    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx8jw\" (UniqueName: \"kubernetes.io/projected/77cb750c-2c2f-41b9-ada6-7021f53ea271-kube-api-access-vx8jw\") pod \"console-65b6bfd656-g7zrp\" (UID: \"77cb750c-2c2f-41b9-ada6-7021f53ea271\") " pod="openshift-console/console-65b6bfd656-g7zrp"
Jun 15 05:49:00.112529 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:00.112496    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65b6bfd656-g7zrp"
Jun 15 05:49:00.904341 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:00.904067    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65b6bfd656-g7zrp"]
Jun 15 05:49:00.908569 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:49:00.908449    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77cb750c_2c2f_41b9_ada6_7021f53ea271.slice/crio-de1126c4839f05229fdc5d4e841d684c7f9d2f247f47f72cc61a56e8a43fc147 WatchSource:0}: Error finding container de1126c4839f05229fdc5d4e841d684c7f9d2f247f47f72cc61a56e8a43fc147: Status 404 returned error can't find the container with id de1126c4839f05229fdc5d4e841d684c7f9d2f247f47f72cc61a56e8a43fc147
Jun 15 05:49:00.920447 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:00.920031    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-864fcfc6f9-54pzw"]
Jun 15 05:49:00.927549 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:49:00.927513    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96f776da_6137_4f8b_8324_e6cab29e36d6.slice/crio-272e3ccfe6c072ab4e864335a714b424889b851906e62ba2336ac342ffa9feae WatchSource:0}: Error finding container 272e3ccfe6c072ab4e864335a714b424889b851906e62ba2336ac342ffa9feae: Status 404 returned error can't find the container with id 272e3ccfe6c072ab4e864335a714b424889b851906e62ba2336ac342ffa9feae
Jun 15 05:49:01.704947 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:01.704236    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xpdg2" event={"ID":"bdbef6be-323c-4a82-9f0a-8bb8fdafc620","Type":"ContainerStarted","Data":"221ccacac0b9328efa51a5b6073a0021117bae1fed7412ae5a50ce27d051a609"}
Jun 15 05:49:01.705159 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:01.705052    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xpdg2" event={"ID":"bdbef6be-323c-4a82-9f0a-8bb8fdafc620","Type":"ContainerStarted","Data":"8194f9aa9ff01c8eed6bb283862d68b9d3953b444c831cd7de56c9db3e03e83c"}
Jun 15 05:49:01.705820 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:01.705793    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65b6bfd656-g7zrp" event={"ID":"77cb750c-2c2f-41b9-ada6-7021f53ea271","Type":"ContainerStarted","Data":"de1126c4839f05229fdc5d4e841d684c7f9d2f247f47f72cc61a56e8a43fc147"}
Jun 15 05:49:01.707611 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:01.707586    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-k5l48" event={"ID":"6f7e3592-7192-4acb-a079-7cc24646f8a5","Type":"ContainerStarted","Data":"16a9d96b0c5ae1cdbae6733651aff3c5efed488cfa39e91c9762ec85d4b83830"}
Jun 15 05:49:01.709969 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:01.709940    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q6c87" event={"ID":"47e37b20-ad06-4ff6-9beb-6503464f6ef8","Type":"ContainerStarted","Data":"69da7f0c91ddcf29f0a81e22c2c029b3f38ca715c6dcefe841f8df5f0144b513"}
Jun 15 05:49:01.710090 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:01.709988    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q6c87" event={"ID":"47e37b20-ad06-4ff6-9beb-6503464f6ef8","Type":"ContainerStarted","Data":"b11a1f5318edab2085fde75d66c1d47956b24b9e79dd55eeff34145d926d09a7"}
Jun 15 05:49:01.710090 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:01.710039    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-q6c87"
Jun 15 05:49:01.711376 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:01.711342    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-864fcfc6f9-54pzw" event={"ID":"96f776da-6137-4f8b-8324-e6cab29e36d6","Type":"ContainerStarted","Data":"272e3ccfe6c072ab4e864335a714b424889b851906e62ba2336ac342ffa9feae"}
Jun 15 05:49:01.738544 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:01.738297    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-k5l48" podStartSLOduration=1.768954845 podStartE2EDuration="6.738277455s" totalImagesPullingTime="4.96932261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:48:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:01.737796126 +0000 UTC m=+56.885976124" watchObservedRunningTime="2026-06-15 05:49:01.738277455 +0000 UTC m=+56.886457433"
Jun 15 05:49:01.763205 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:01.763083    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-q6c87" podStartSLOduration=1.791414796 podStartE2EDuration="6.763056863s" totalImagesPullingTime="4.971642067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:48:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:01.761472561 +0000 UTC m=+56.909652559" watchObservedRunningTime="2026-06-15 05:49:01.763056863 +0000 UTC m=+56.911236866"
Jun 15 05:49:02.717624 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:02.717588    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-864fcfc6f9-54pzw" event={"ID":"96f776da-6137-4f8b-8324-e6cab29e36d6","Type":"ContainerStarted","Data":"ddbde43c3a547f35fcb6e892a7637d72bc2e922058bdb576a766ddeb1589c1d8"}
Jun 15 05:49:02.718157 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:02.717655    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-864fcfc6f9-54pzw" event={"ID":"96f776da-6137-4f8b-8324-e6cab29e36d6","Type":"ContainerStarted","Data":"fab74404891cd89d043bb47e7af219228ca559ad35a9fe6c97b651dd4deed218"}
Jun 15 05:49:02.743223 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:02.743159    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-864fcfc6f9-54pzw" podStartSLOduration=4.537949485 podStartE2EDuration="5.743137388s" totalImagesPullingTime="1.205187903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:48:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:02.742479406 +0000 UTC m=+57.890659403" watchObservedRunningTime="2026-06-15 05:49:02.743137388 +0000 UTC m=+57.891317395"
Jun 15 05:49:04.542500 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.542462    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-6b8d965dd5-x6l2f"]
Jun 15 05:49:04.544568 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.544550    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-x6l2f"
Jun 15 05:49:04.554598 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.554523    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jun 15 05:49:04.555372 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.555333    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jun 15 05:49:04.555457 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.555371    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-5s2qv\""
Jun 15 05:49:04.572263 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.572230    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6b8d965dd5-x6l2f"]
Jun 15 05:49:04.587604 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.587572    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-678bc549d7-7746w"]
Jun 15 05:49:04.589650 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.589636    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-678bc549d7-7746w"
Jun 15 05:49:04.593430 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.593409    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jun 15 05:49:04.593991 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.593975    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jun 15 05:49:04.594089 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.593977    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-nz24d\""
Jun 15 05:49:04.594945 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.594926    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jun 15 05:49:04.610142 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.610120    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-678bc549d7-7746w"]
Jun 15 05:49:04.642268 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.639391    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptxhh\" (UniqueName: \"kubernetes.io/projected/c7af7387-0238-4b94-9a45-8bcaa37446fd-kube-api-access-ptxhh\") pod \"kube-state-metrics-678bc549d7-7746w\" (UID: \"c7af7387-0238-4b94-9a45-8bcaa37446fd\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-7746w"
Jun 15 05:49:04.642268 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.639459    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c7af7387-0238-4b94-9a45-8bcaa37446fd-volume-directive-shadow\") pod \"kube-state-metrics-678bc549d7-7746w\" (UID: \"c7af7387-0238-4b94-9a45-8bcaa37446fd\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-7746w"
Jun 15 05:49:04.642268 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.639493    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7af7387-0238-4b94-9a45-8bcaa37446fd-metrics-client-ca\") pod \"kube-state-metrics-678bc549d7-7746w\" (UID: \"c7af7387-0238-4b94-9a45-8bcaa37446fd\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-7746w"
Jun 15 05:49:04.642268 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.639547    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c7af7387-0238-4b94-9a45-8bcaa37446fd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-678bc549d7-7746w\" (UID: \"c7af7387-0238-4b94-9a45-8bcaa37446fd\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-7746w"
Jun 15 05:49:04.642268 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.639580    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c7af7387-0238-4b94-9a45-8bcaa37446fd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-678bc549d7-7746w\" (UID: \"c7af7387-0238-4b94-9a45-8bcaa37446fd\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-7746w"
Jun 15 05:49:04.642268 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.639613    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17ffae6f-c8ca-4b19-bd6a-20a89393597b-metrics-client-ca\") pod \"openshift-state-metrics-6b8d965dd5-x6l2f\" (UID: \"17ffae6f-c8ca-4b19-bd6a-20a89393597b\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-x6l2f"
Jun 15 05:49:04.642268 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.639665    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/17ffae6f-c8ca-4b19-bd6a-20a89393597b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6b8d965dd5-x6l2f\" (UID: \"17ffae6f-c8ca-4b19-bd6a-20a89393597b\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-x6l2f"
Jun 15 05:49:04.642268 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.639698    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c7af7387-0238-4b94-9a45-8bcaa37446fd-kube-state-metrics-tls\") pod \"kube-state-metrics-678bc549d7-7746w\" (UID: \"c7af7387-0238-4b94-9a45-8bcaa37446fd\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-7746w"
Jun 15 05:49:04.642268 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.639889    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfb96\" (UniqueName: \"kubernetes.io/projected/17ffae6f-c8ca-4b19-bd6a-20a89393597b-kube-api-access-rfb96\") pod \"openshift-state-metrics-6b8d965dd5-x6l2f\" (UID: \"17ffae6f-c8ca-4b19-bd6a-20a89393597b\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-x6l2f"
Jun 15 05:49:04.642268 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.639956    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/17ffae6f-c8ca-4b19-bd6a-20a89393597b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6b8d965dd5-x6l2f\" (UID: \"17ffae6f-c8ca-4b19-bd6a-20a89393597b\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-x6l2f"
Jun 15 05:49:04.677926 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.677894    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9hzhf"]
Jun 15 05:49:04.680054 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.680039    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:04.695553 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.695530    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jun 15 05:49:04.695694 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.695678    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5jhpc\""
Jun 15 05:49:04.696008 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.695995    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jun 15 05:49:04.698173 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.698158    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jun 15 05:49:04.725708 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.725678    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-xpdg2" event={"ID":"bdbef6be-323c-4a82-9f0a-8bb8fdafc620","Type":"ContainerStarted","Data":"4707ddcd55b50bb209446ec543994db093495c5b96200605a26cb1204ece34d7"}
Jun 15 05:49:04.727179 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.727160    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65b6bfd656-g7zrp" event={"ID":"77cb750c-2c2f-41b9-ada6-7021f53ea271","Type":"ContainerStarted","Data":"fcfcb3b9d2cc725fa3840803da87a3af8171ac7641ba6503576e00f41d81d4bf"}
Jun 15 05:49:04.741150 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.741125    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfb96\" (UniqueName: \"kubernetes.io/projected/17ffae6f-c8ca-4b19-bd6a-20a89393597b-kube-api-access-rfb96\") pod \"openshift-state-metrics-6b8d965dd5-x6l2f\" (UID: \"17ffae6f-c8ca-4b19-bd6a-20a89393597b\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-x6l2f"
Jun 15 05:49:04.741246 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.741156    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24202341-f48a-4a16-9619-eab661b11dc3-metrics-client-ca\") pod \"node-exporter-9hzhf\" (UID: \"24202341-f48a-4a16-9619-eab661b11dc3\") " pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:04.741246 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.741176    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/17ffae6f-c8ca-4b19-bd6a-20a89393597b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6b8d965dd5-x6l2f\" (UID: \"17ffae6f-c8ca-4b19-bd6a-20a89393597b\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-x6l2f"
Jun 15 05:49:04.741246 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.741197    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24202341-f48a-4a16-9619-eab661b11dc3-sys\") pod \"node-exporter-9hzhf\" (UID: \"24202341-f48a-4a16-9619-eab661b11dc3\") " pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:04.741341 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.741246    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptxhh\" (UniqueName: \"kubernetes.io/projected/c7af7387-0238-4b94-9a45-8bcaa37446fd-kube-api-access-ptxhh\") pod \"kube-state-metrics-678bc549d7-7746w\" (UID: \"c7af7387-0238-4b94-9a45-8bcaa37446fd\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-7746w"
Jun 15 05:49:04.741341 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:49:04.741290    2572 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jun 15 05:49:04.741341 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.741308    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4sdp\" (UniqueName: \"kubernetes.io/projected/24202341-f48a-4a16-9619-eab661b11dc3-kube-api-access-b4sdp\") pod \"node-exporter-9hzhf\" (UID: \"24202341-f48a-4a16-9619-eab661b11dc3\") " pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:04.741341 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.741335    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c7af7387-0238-4b94-9a45-8bcaa37446fd-volume-directive-shadow\") pod \"kube-state-metrics-678bc549d7-7746w\" (UID: \"c7af7387-0238-4b94-9a45-8bcaa37446fd\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-7746w"
Jun 15 05:49:04.741521 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.741353    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7af7387-0238-4b94-9a45-8bcaa37446fd-metrics-client-ca\") pod \"kube-state-metrics-678bc549d7-7746w\" (UID: \"c7af7387-0238-4b94-9a45-8bcaa37446fd\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-7746w"
Jun 15 05:49:04.741521 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:49:04.741404    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17ffae6f-c8ca-4b19-bd6a-20a89393597b-openshift-state-metrics-tls podName:17ffae6f-c8ca-4b19-bd6a-20a89393597b nodeName:}" failed. No retries permitted until 2026-06-15 05:49:05.241384521 +0000 UTC m=+60.389564515 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/17ffae6f-c8ca-4b19-bd6a-20a89393597b-openshift-state-metrics-tls") pod "openshift-state-metrics-6b8d965dd5-x6l2f" (UID: "17ffae6f-c8ca-4b19-bd6a-20a89393597b") : secret "openshift-state-metrics-tls" not found
Jun 15 05:49:04.741521 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.741459    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/24202341-f48a-4a16-9619-eab661b11dc3-root\") pod \"node-exporter-9hzhf\" (UID: \"24202341-f48a-4a16-9619-eab661b11dc3\") " pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:04.741521 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.741497    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/24202341-f48a-4a16-9619-eab661b11dc3-node-exporter-wtmp\") pod \"node-exporter-9hzhf\" (UID: \"24202341-f48a-4a16-9619-eab661b11dc3\") " pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:04.741676 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.741523    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/24202341-f48a-4a16-9619-eab661b11dc3-node-exporter-accelerators-collector-config\") pod \"node-exporter-9hzhf\" (UID: \"24202341-f48a-4a16-9619-eab661b11dc3\") " pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:04.741676 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.741559    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c7af7387-0238-4b94-9a45-8bcaa37446fd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-678bc549d7-7746w\" (UID: \"c7af7387-0238-4b94-9a45-8bcaa37446fd\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-7746w"
Jun 15 05:49:04.741676 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.741581    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c7af7387-0238-4b94-9a45-8bcaa37446fd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-678bc549d7-7746w\" (UID: \"c7af7387-0238-4b94-9a45-8bcaa37446fd\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-7746w"
Jun 15 05:49:04.741676 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.741600    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/24202341-f48a-4a16-9619-eab661b11dc3-node-exporter-tls\") pod \"node-exporter-9hzhf\" (UID: \"24202341-f48a-4a16-9619-eab661b11dc3\") " pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:04.741676 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.741632    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17ffae6f-c8ca-4b19-bd6a-20a89393597b-metrics-client-ca\") pod \"openshift-state-metrics-6b8d965dd5-x6l2f\" (UID: \"17ffae6f-c8ca-4b19-bd6a-20a89393597b\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-x6l2f"
Jun 15 05:49:04.741850 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.741741    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c7af7387-0238-4b94-9a45-8bcaa37446fd-volume-directive-shadow\") pod \"kube-state-metrics-678bc549d7-7746w\" (UID: \"c7af7387-0238-4b94-9a45-8bcaa37446fd\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-7746w"
Jun 15 05:49:04.741850 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.741811    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/17ffae6f-c8ca-4b19-bd6a-20a89393597b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6b8d965dd5-x6l2f\" (UID: \"17ffae6f-c8ca-4b19-bd6a-20a89393597b\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-x6l2f"
Jun 15 05:49:04.741850 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.741840    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/24202341-f48a-4a16-9619-eab661b11dc3-node-exporter-textfile\") pod \"node-exporter-9hzhf\" (UID: \"24202341-f48a-4a16-9619-eab661b11dc3\") " pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:04.742139 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.741870    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c7af7387-0238-4b94-9a45-8bcaa37446fd-kube-state-metrics-tls\") pod \"kube-state-metrics-678bc549d7-7746w\" (UID: \"c7af7387-0238-4b94-9a45-8bcaa37446fd\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-7746w"
Jun 15 05:49:04.742139 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.741939    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24202341-f48a-4a16-9619-eab661b11dc3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9hzhf\" (UID: \"24202341-f48a-4a16-9619-eab661b11dc3\") " pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:04.742259 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.742178    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7af7387-0238-4b94-9a45-8bcaa37446fd-metrics-client-ca\") pod \"kube-state-metrics-678bc549d7-7746w\" (UID: \"c7af7387-0238-4b94-9a45-8bcaa37446fd\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-7746w"
Jun 15 05:49:04.742314 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.742257    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c7af7387-0238-4b94-9a45-8bcaa37446fd-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-678bc549d7-7746w\" (UID: \"c7af7387-0238-4b94-9a45-8bcaa37446fd\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-7746w"
Jun 15 05:49:04.742464 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.742439    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17ffae6f-c8ca-4b19-bd6a-20a89393597b-metrics-client-ca\") pod \"openshift-state-metrics-6b8d965dd5-x6l2f\" (UID: \"17ffae6f-c8ca-4b19-bd6a-20a89393597b\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-x6l2f"
Jun 15 05:49:04.744175 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.744143    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c7af7387-0238-4b94-9a45-8bcaa37446fd-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-678bc549d7-7746w\" (UID: \"c7af7387-0238-4b94-9a45-8bcaa37446fd\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-7746w"
Jun 15 05:49:04.744252 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.744198    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c7af7387-0238-4b94-9a45-8bcaa37446fd-kube-state-metrics-tls\") pod \"kube-state-metrics-678bc549d7-7746w\" (UID: \"c7af7387-0238-4b94-9a45-8bcaa37446fd\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-7746w"
Jun 15 05:49:04.744252 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.744200    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/17ffae6f-c8ca-4b19-bd6a-20a89393597b-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-6b8d965dd5-x6l2f\" (UID: \"17ffae6f-c8ca-4b19-bd6a-20a89393597b\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-x6l2f"
Jun 15 05:49:04.765570 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.765550    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfb96\" (UniqueName: \"kubernetes.io/projected/17ffae6f-c8ca-4b19-bd6a-20a89393597b-kube-api-access-rfb96\") pod \"openshift-state-metrics-6b8d965dd5-x6l2f\" (UID: \"17ffae6f-c8ca-4b19-bd6a-20a89393597b\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-x6l2f"
Jun 15 05:49:04.770749 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.770720    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptxhh\" (UniqueName: \"kubernetes.io/projected/c7af7387-0238-4b94-9a45-8bcaa37446fd-kube-api-access-ptxhh\") pod \"kube-state-metrics-678bc549d7-7746w\" (UID: \"c7af7387-0238-4b94-9a45-8bcaa37446fd\") " pod="openshift-monitoring/kube-state-metrics-678bc549d7-7746w"
Jun 15 05:49:04.817838 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.817715    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65b6bfd656-g7zrp" podStartSLOduration=2.4217335269999998 podStartE2EDuration="5.81769815s" totalImagesPullingTime="3.395964623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:48:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:04.816580244 +0000 UTC m=+59.964760240" watchObservedRunningTime="2026-06-15 05:49:04.81769815 +0000 UTC m=+59.965878147"
Jun 15 05:49:04.842675 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.842642    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/24202341-f48a-4a16-9619-eab661b11dc3-node-exporter-accelerators-collector-config\") pod \"node-exporter-9hzhf\" (UID: \"24202341-f48a-4a16-9619-eab661b11dc3\") " pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:04.842889 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.842864    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/24202341-f48a-4a16-9619-eab661b11dc3-node-exporter-tls\") pod \"node-exporter-9hzhf\" (UID: \"24202341-f48a-4a16-9619-eab661b11dc3\") " pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:04.843626 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.842934    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/24202341-f48a-4a16-9619-eab661b11dc3-node-exporter-textfile\") pod \"node-exporter-9hzhf\" (UID: \"24202341-f48a-4a16-9619-eab661b11dc3\") " pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:04.843626 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.842983    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24202341-f48a-4a16-9619-eab661b11dc3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9hzhf\" (UID: \"24202341-f48a-4a16-9619-eab661b11dc3\") " pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:04.843626 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.843060    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24202341-f48a-4a16-9619-eab661b11dc3-metrics-client-ca\") pod \"node-exporter-9hzhf\" (UID: \"24202341-f48a-4a16-9619-eab661b11dc3\") " pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:04.843626 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:49:04.843060    2572 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jun 15 05:49:04.843626 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.843173    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24202341-f48a-4a16-9619-eab661b11dc3-sys\") pod \"node-exporter-9hzhf\" (UID: \"24202341-f48a-4a16-9619-eab661b11dc3\") " pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:04.843626 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.843219    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4sdp\" (UniqueName: \"kubernetes.io/projected/24202341-f48a-4a16-9619-eab661b11dc3-kube-api-access-b4sdp\") pod \"node-exporter-9hzhf\" (UID: \"24202341-f48a-4a16-9619-eab661b11dc3\") " pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:04.843626 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.843252    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/24202341-f48a-4a16-9619-eab661b11dc3-root\") pod \"node-exporter-9hzhf\" (UID: \"24202341-f48a-4a16-9619-eab661b11dc3\") " pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:04.843626 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.843254    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/24202341-f48a-4a16-9619-eab661b11dc3-node-exporter-accelerators-collector-config\") pod \"node-exporter-9hzhf\" (UID: \"24202341-f48a-4a16-9619-eab661b11dc3\") " pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:04.843626 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.843277    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/24202341-f48a-4a16-9619-eab661b11dc3-node-exporter-wtmp\") pod \"node-exporter-9hzhf\" (UID: \"24202341-f48a-4a16-9619-eab661b11dc3\") " pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:04.843626 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.843296    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/24202341-f48a-4a16-9619-eab661b11dc3-node-exporter-textfile\") pod \"node-exporter-9hzhf\" (UID: \"24202341-f48a-4a16-9619-eab661b11dc3\") " pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:04.843626 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.843382    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/24202341-f48a-4a16-9619-eab661b11dc3-sys\") pod \"node-exporter-9hzhf\" (UID: \"24202341-f48a-4a16-9619-eab661b11dc3\") " pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:04.843626 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.843433    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/24202341-f48a-4a16-9619-eab661b11dc3-node-exporter-wtmp\") pod \"node-exporter-9hzhf\" (UID: \"24202341-f48a-4a16-9619-eab661b11dc3\") " pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:04.843626 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:49:04.843447    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24202341-f48a-4a16-9619-eab661b11dc3-node-exporter-tls podName:24202341-f48a-4a16-9619-eab661b11dc3 nodeName:}" failed. No retries permitted until 2026-06-15 05:49:05.343426837 +0000 UTC m=+60.491606819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/24202341-f48a-4a16-9619-eab661b11dc3-node-exporter-tls") pod "node-exporter-9hzhf" (UID: "24202341-f48a-4a16-9619-eab661b11dc3") : secret "node-exporter-tls" not found
Jun 15 05:49:04.843626 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.843502    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/24202341-f48a-4a16-9619-eab661b11dc3-root\") pod \"node-exporter-9hzhf\" (UID: \"24202341-f48a-4a16-9619-eab661b11dc3\") " pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:04.844113 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.843785    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/24202341-f48a-4a16-9619-eab661b11dc3-metrics-client-ca\") pod \"node-exporter-9hzhf\" (UID: \"24202341-f48a-4a16-9619-eab661b11dc3\") " pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:04.845703 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.845685    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/24202341-f48a-4a16-9619-eab661b11dc3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9hzhf\" (UID: \"24202341-f48a-4a16-9619-eab661b11dc3\") " pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:04.852261 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.852217    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-xpdg2" podStartSLOduration=1.550732994 podStartE2EDuration="9.852204166s" totalImagesPullingTime="8.301471172s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:48:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:04.847523471 +0000 UTC m=+59.995703467" watchObservedRunningTime="2026-06-15 05:49:04.852204166 +0000 UTC m=+60.000384163"
Jun 15 05:49:04.854206 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.854181    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4sdp\" (UniqueName: \"kubernetes.io/projected/24202341-f48a-4a16-9619-eab661b11dc3-kube-api-access-b4sdp\") pod \"node-exporter-9hzhf\" (UID: \"24202341-f48a-4a16-9619-eab661b11dc3\") " pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:04.898259 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:04.898227    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-678bc549d7-7746w"
Jun 15 05:49:05.024116 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.024083    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-678bc549d7-7746w"]
Jun 15 05:49:05.026998 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:49:05.026975    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7af7387_0238_4b94_9a45_8bcaa37446fd.slice/crio-77082f10da1d1560c50238d3a5ac4522501fef09e2907773cfbba5960f180ccb WatchSource:0}: Error finding container 77082f10da1d1560c50238d3a5ac4522501fef09e2907773cfbba5960f180ccb: Status 404 returned error can't find the container with id 77082f10da1d1560c50238d3a5ac4522501fef09e2907773cfbba5960f180ccb
Jun 15 05:49:05.247400 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.246666    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/17ffae6f-c8ca-4b19-bd6a-20a89393597b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6b8d965dd5-x6l2f\" (UID: \"17ffae6f-c8ca-4b19-bd6a-20a89393597b\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-x6l2f"
Jun 15 05:49:05.247400 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:49:05.246839    2572 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jun 15 05:49:05.247400 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:49:05.246892    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17ffae6f-c8ca-4b19-bd6a-20a89393597b-openshift-state-metrics-tls podName:17ffae6f-c8ca-4b19-bd6a-20a89393597b nodeName:}" failed. No retries permitted until 2026-06-15 05:49:06.246874849 +0000 UTC m=+61.395054826 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/17ffae6f-c8ca-4b19-bd6a-20a89393597b-openshift-state-metrics-tls") pod "openshift-state-metrics-6b8d965dd5-x6l2f" (UID: "17ffae6f-c8ca-4b19-bd6a-20a89393597b") : secret "openshift-state-metrics-tls" not found
Jun 15 05:49:05.347628 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.347600    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/24202341-f48a-4a16-9619-eab661b11dc3-node-exporter-tls\") pod \"node-exporter-9hzhf\" (UID: \"24202341-f48a-4a16-9619-eab661b11dc3\") " pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:05.349739 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.349709    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/24202341-f48a-4a16-9619-eab661b11dc3-node-exporter-tls\") pod \"node-exporter-9hzhf\" (UID: \"24202341-f48a-4a16-9619-eab661b11dc3\") " pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:05.591051 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.590975    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-5jhpc\""
Jun 15 05:49:05.599050 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.599024    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9hzhf"
Jun 15 05:49:05.610613 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:49:05.610581    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24202341_f48a_4a16_9619_eab661b11dc3.slice/crio-6f33b19d59cc87d6c03bbbb389ad4609a6a6c0364bc99a3ba3832850361de326 WatchSource:0}: Error finding container 6f33b19d59cc87d6c03bbbb389ad4609a6a6c0364bc99a3ba3832850361de326: Status 404 returned error can't find the container with id 6f33b19d59cc87d6c03bbbb389ad4609a6a6c0364bc99a3ba3832850361de326
Jun 15 05:49:05.633241 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.633204    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-98qt9"
Jun 15 05:49:05.688034 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.688003    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jun 15 05:49:05.693912 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.693887    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.696273 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.696242    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jun 15 05:49:05.696445 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.696403    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jun 15 05:49:05.696445 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.696433    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jun 15 05:49:05.696746 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.696712    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-n66l2\""
Jun 15 05:49:05.696746 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.696743    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jun 15 05:49:05.696886 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.696826    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jun 15 05:49:05.696886 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.696831    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jun 15 05:49:05.697101 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.697085    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jun 15 05:49:05.697189 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.697107    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jun 15 05:49:05.701096 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.701075    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jun 15 05:49:05.708557 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.708534    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jun 15 05:49:05.731671 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.731635    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9hzhf" event={"ID":"24202341-f48a-4a16-9619-eab661b11dc3","Type":"ContainerStarted","Data":"6f33b19d59cc87d6c03bbbb389ad4609a6a6c0364bc99a3ba3832850361de326"}
Jun 15 05:49:05.733013 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.732985    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-678bc549d7-7746w" event={"ID":"c7af7387-0238-4b94-9a45-8bcaa37446fd","Type":"ContainerStarted","Data":"77082f10da1d1560c50238d3a5ac4522501fef09e2907773cfbba5960f180ccb"}
Jun 15 05:49:05.749488 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.749461    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfzxh\" (UniqueName: \"kubernetes.io/projected/08a5d13e-ee76-4cd0-b100-e51a330f1765-kube-api-access-rfzxh\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.749655 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.749498    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08a5d13e-ee76-4cd0-b100-e51a330f1765-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.749655 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.749525    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.749655 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.749596    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.749817 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.749663    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/08a5d13e-ee76-4cd0-b100-e51a330f1765-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.749817 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.749805    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.749920 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.749836    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-config-volume\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.749920 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.749876    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08a5d13e-ee76-4cd0-b100-e51a330f1765-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.750008 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.749922    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/08a5d13e-ee76-4cd0-b100-e51a330f1765-tls-assets\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.750008 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.749955    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-web-config\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.750121 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.750084    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.750196 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.750160    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/08a5d13e-ee76-4cd0-b100-e51a330f1765-config-out\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.750281 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.750254    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.851317 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.851230    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08a5d13e-ee76-4cd0-b100-e51a330f1765-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.851317 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.851272    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/08a5d13e-ee76-4cd0-b100-e51a330f1765-tls-assets\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.851579 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.851414    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-web-config\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.851579 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.851488    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.851752 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.851726    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/08a5d13e-ee76-4cd0-b100-e51a330f1765-config-out\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.851826 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.851773    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.851883 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.851828    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfzxh\" (UniqueName: \"kubernetes.io/projected/08a5d13e-ee76-4cd0-b100-e51a330f1765-kube-api-access-rfzxh\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.851883 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.851857    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08a5d13e-ee76-4cd0-b100-e51a330f1765-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.851978 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.851891    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.851978 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.851939    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.852076 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.851976    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/08a5d13e-ee76-4cd0-b100-e51a330f1765-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.852076 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.852015    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08a5d13e-ee76-4cd0-b100-e51a330f1765-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.852173 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.852153    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.852240 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.852185    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-config-volume\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.853242 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.852786    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08a5d13e-ee76-4cd0-b100-e51a330f1765-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.853242 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.853090    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/08a5d13e-ee76-4cd0-b100-e51a330f1765-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.854406 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.854381    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/08a5d13e-ee76-4cd0-b100-e51a330f1765-tls-assets\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.854916 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.854875    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-web-config\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.855627 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.855608    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/08a5d13e-ee76-4cd0-b100-e51a330f1765-config-out\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.855955 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.855935    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.856020 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.855985    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.856451 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.856433    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.856538 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.856458    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.856538 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.856504    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-config-volume\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.856538 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.856520    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:05.861259 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:05.861233    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfzxh\" (UniqueName: \"kubernetes.io/projected/08a5d13e-ee76-4cd0-b100-e51a330f1765-kube-api-access-rfzxh\") pod \"alertmanager-main-0\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:06.004866 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:06.004829    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:49:06.255370 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:06.255329    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/17ffae6f-c8ca-4b19-bd6a-20a89393597b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6b8d965dd5-x6l2f\" (UID: \"17ffae6f-c8ca-4b19-bd6a-20a89393597b\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-x6l2f"
Jun 15 05:49:06.257759 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:06.257733    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/17ffae6f-c8ca-4b19-bd6a-20a89393597b-openshift-state-metrics-tls\") pod \"openshift-state-metrics-6b8d965dd5-x6l2f\" (UID: \"17ffae6f-c8ca-4b19-bd6a-20a89393597b\") " pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-x6l2f"
Jun 15 05:49:06.355693 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:06.355663    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-5s2qv\""
Jun 15 05:49:06.363761 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:06.363737    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-x6l2f"
Jun 15 05:49:06.370144 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:06.370036    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jun 15 05:49:06.505276 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:06.505206    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-6b8d965dd5-x6l2f"]
Jun 15 05:49:06.594702 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:49:06.594661    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ffae6f_c8ca_4b19_bd6a_20a89393597b.slice/crio-1bd4d0610a8ce438614790869fc1ddb107c40278d96eb0d48b3006b8ae834233 WatchSource:0}: Error finding container 1bd4d0610a8ce438614790869fc1ddb107c40278d96eb0d48b3006b8ae834233: Status 404 returned error can't find the container with id 1bd4d0610a8ce438614790869fc1ddb107c40278d96eb0d48b3006b8ae834233
Jun 15 05:49:06.739553 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:06.739519    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-678bc549d7-7746w" event={"ID":"c7af7387-0238-4b94-9a45-8bcaa37446fd","Type":"ContainerStarted","Data":"4678449a2ba4b19cac3d48a5f48b0e83efb9c07d68c69f14b69da75ad0fbd408"}
Jun 15 05:49:06.741725 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:06.741698    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-x6l2f" event={"ID":"17ffae6f-c8ca-4b19-bd6a-20a89393597b","Type":"ContainerStarted","Data":"848c7699d6181fe15275185ea74935182c2a4789a4843dda07b14062d3fa6519"}
Jun 15 05:49:06.741807 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:06.741759    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-x6l2f" event={"ID":"17ffae6f-c8ca-4b19-bd6a-20a89393597b","Type":"ContainerStarted","Data":"1bd4d0610a8ce438614790869fc1ddb107c40278d96eb0d48b3006b8ae834233"}
Jun 15 05:49:06.743273 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:06.743246    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a5d13e-ee76-4cd0-b100-e51a330f1765","Type":"ContainerStarted","Data":"f4f15bf77a99aea77517191c5c7c2f3e0f428fc2aa1417140b4921b2af839c9f"}
Jun 15 05:49:07.544769 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.544737    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-569bcc98cc-w6242"]
Jun 15 05:49:07.547545 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.547522    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242"
Jun 15 05:49:07.550489 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.550463    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jun 15 05:49:07.550787 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.550764    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-99ahvvetk7vfp\""
Jun 15 05:49:07.551764 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.551743    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-xlskg\""
Jun 15 05:49:07.552621 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.552529    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jun 15 05:49:07.552718 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.552665    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jun 15 05:49:07.554533 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.554519    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jun 15 05:49:07.557115 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.556973    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jun 15 05:49:07.577349 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.577261    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-569bcc98cc-w6242"]
Jun 15 05:49:07.668665 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.668625    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-569bcc98cc-w6242\" (UID: \"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef\") " pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242"
Jun 15 05:49:07.668665 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.668667    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-569bcc98cc-w6242\" (UID: \"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef\") " pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242"
Jun 15 05:49:07.669146 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.668694    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsw6x\" (UniqueName: \"kubernetes.io/projected/d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef-kube-api-access-hsw6x\") pod \"thanos-querier-569bcc98cc-w6242\" (UID: \"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef\") " pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242"
Jun 15 05:49:07.669146 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.668717    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef-metrics-client-ca\") pod \"thanos-querier-569bcc98cc-w6242\" (UID: \"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef\") " pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242"
Jun 15 05:49:07.669146 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.668741    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-569bcc98cc-w6242\" (UID: \"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef\") " pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242"
Jun 15 05:49:07.669146 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.668796    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-569bcc98cc-w6242\" (UID: \"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef\") " pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242"
Jun 15 05:49:07.669146 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.668870    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef-secret-grpc-tls\") pod \"thanos-querier-569bcc98cc-w6242\" (UID: \"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef\") " pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242"
Jun 15 05:49:07.669146 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.668939    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef-secret-thanos-querier-tls\") pod \"thanos-querier-569bcc98cc-w6242\" (UID: \"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef\") " pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242"
Jun 15 05:49:07.750321 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.750278    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9hzhf" event={"ID":"24202341-f48a-4a16-9619-eab661b11dc3","Type":"ContainerStarted","Data":"25d72214ec30e5118f335fb6e6fd9745e853ae16e1d370eec392499b5478c7c3"}
Jun 15 05:49:07.752855 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.752825    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-678bc549d7-7746w" event={"ID":"c7af7387-0238-4b94-9a45-8bcaa37446fd","Type":"ContainerStarted","Data":"64829c975e4e9b26d27b8e973f51c700b9877a2123a22df3e3e75214c6524733"}
Jun 15 05:49:07.752991 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.752888    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-678bc549d7-7746w" event={"ID":"c7af7387-0238-4b94-9a45-8bcaa37446fd","Type":"ContainerStarted","Data":"ec99e33653c4c63619a940e3c457cc14ead14c75c500074be75805c99dfcb9e9"}
Jun 15 05:49:07.755060 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.755031    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-x6l2f" event={"ID":"17ffae6f-c8ca-4b19-bd6a-20a89393597b","Type":"ContainerStarted","Data":"a6c1f3ffad99863fbdbe9febfd50c5cf8f0336734d6bb0f349008af33f90da97"}
Jun 15 05:49:07.770374 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.770340    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-569bcc98cc-w6242\" (UID: \"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef\") " pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242"
Jun 15 05:49:07.770493 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.770390    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-569bcc98cc-w6242\" (UID: \"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef\") " pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242"
Jun 15 05:49:07.770493 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.770417    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsw6x\" (UniqueName: \"kubernetes.io/projected/d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef-kube-api-access-hsw6x\") pod \"thanos-querier-569bcc98cc-w6242\" (UID: \"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef\") " pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242"
Jun 15 05:49:07.770493 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.770441    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef-metrics-client-ca\") pod \"thanos-querier-569bcc98cc-w6242\" (UID: \"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef\") " pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242"
Jun 15 05:49:07.770493 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.770467    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-569bcc98cc-w6242\" (UID: \"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef\") " pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242"
Jun 15 05:49:07.770781 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.770730    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-569bcc98cc-w6242\" (UID: \"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef\") " pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242"
Jun 15 05:49:07.770880 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.770785    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef-secret-grpc-tls\") pod \"thanos-querier-569bcc98cc-w6242\" (UID: \"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef\") " pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242"
Jun 15 05:49:07.770880 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.770843    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef-secret-thanos-querier-tls\") pod \"thanos-querier-569bcc98cc-w6242\" (UID: \"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef\") " pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242"
Jun 15 05:49:07.772323 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.772301    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef-metrics-client-ca\") pod \"thanos-querier-569bcc98cc-w6242\" (UID: \"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef\") " pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242"
Jun 15 05:49:07.774261 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.774138    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-569bcc98cc-w6242\" (UID: \"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef\") " pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242"
Jun 15 05:49:07.775512 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.774790    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef-secret-grpc-tls\") pod \"thanos-querier-569bcc98cc-w6242\" (UID: \"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef\") " pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242"
Jun 15 05:49:07.775512 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.774942    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef-secret-thanos-querier-tls\") pod \"thanos-querier-569bcc98cc-w6242\" (UID: \"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef\") " pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242"
Jun 15 05:49:07.775512 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.775206    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-569bcc98cc-w6242\" (UID: \"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef\") " pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242"
Jun 15 05:49:07.775840 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.775814    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-569bcc98cc-w6242\" (UID: \"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef\") " pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242"
Jun 15 05:49:07.776099 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.776082    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-569bcc98cc-w6242\" (UID: \"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef\") " pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242"
Jun 15 05:49:07.780466 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.780439    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsw6x\" (UniqueName: \"kubernetes.io/projected/d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef-kube-api-access-hsw6x\") pod \"thanos-querier-569bcc98cc-w6242\" (UID: \"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef\") " pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242"
Jun 15 05:49:07.794483 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.794433    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-678bc549d7-7746w" podStartSLOduration=2.204996703 podStartE2EDuration="3.794414819s" totalImagesPullingTime="1.589418116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:49:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:07.792805283 +0000 UTC m=+62.940985303" watchObservedRunningTime="2026-06-15 05:49:07.794414819 +0000 UTC m=+62.942594817"
Jun 15 05:49:07.859841 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:07.859819    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242"
Jun 15 05:49:08.008424 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:08.008385    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-569bcc98cc-w6242"]
Jun 15 05:49:08.012243 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:49:08.012219    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8019fb9_62b7_4c1f_bf6d_c42da66dc6ef.slice/crio-26fbdad2322b2f316bb8de77cef506e176ab8d4de5a23fab8020c7e6ce0ec07f WatchSource:0}: Error finding container 26fbdad2322b2f316bb8de77cef506e176ab8d4de5a23fab8020c7e6ce0ec07f: Status 404 returned error can't find the container with id 26fbdad2322b2f316bb8de77cef506e176ab8d4de5a23fab8020c7e6ce0ec07f
Jun 15 05:49:08.760017 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:08.759981    2572 generic.go:356] "Generic (PLEG): container finished" podID="24202341-f48a-4a16-9619-eab661b11dc3" containerID="25d72214ec30e5118f335fb6e6fd9745e853ae16e1d370eec392499b5478c7c3" exitCode=0
Jun 15 05:49:08.760426 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:08.760064    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9hzhf" event={"ID":"24202341-f48a-4a16-9619-eab661b11dc3","Type":"ContainerDied","Data":"25d72214ec30e5118f335fb6e6fd9745e853ae16e1d370eec392499b5478c7c3"}
Jun 15 05:49:08.762231 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:08.762207    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-x6l2f" event={"ID":"17ffae6f-c8ca-4b19-bd6a-20a89393597b","Type":"ContainerStarted","Data":"28ecb8e299f09007a73f1dadb3769bfbfb4254130d9954c95eea313316958043"}
Jun 15 05:49:08.763799 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:08.763701    2572 generic.go:356] "Generic (PLEG): container finished" podID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerID="2a0e9a629ef24ef07c680061486c63aacc0010e35bdca95013203e691d402900" exitCode=0
Jun 15 05:49:08.763799 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:08.763770    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a5d13e-ee76-4cd0-b100-e51a330f1765","Type":"ContainerDied","Data":"2a0e9a629ef24ef07c680061486c63aacc0010e35bdca95013203e691d402900"}
Jun 15 05:49:08.764999 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:08.764979    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242" event={"ID":"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef","Type":"ContainerStarted","Data":"26fbdad2322b2f316bb8de77cef506e176ab8d4de5a23fab8020c7e6ce0ec07f"}
Jun 15 05:49:08.822434 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:08.822389    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-6b8d965dd5-x6l2f" podStartSLOduration=3.837189648 podStartE2EDuration="4.822369633s" totalImagesPullingTime="985.179985ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:49:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:08.821473618 +0000 UTC m=+63.969653616" watchObservedRunningTime="2026-06-15 05:49:08.822369633 +0000 UTC m=+63.970549624"
Jun 15 05:49:09.187885 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.187849    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-544cd4646d-gzxjx"]
Jun 15 05:49:09.191001 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.190957    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx"
Jun 15 05:49:09.193685 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.193663    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-chj4h\""
Jun 15 05:49:09.194115 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.194059    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bdn3354f4ltrl\""
Jun 15 05:49:09.194488 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.194464    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jun 15 05:49:09.194601 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.194582    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jun 15 05:49:09.194660 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.194470    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jun 15 05:49:09.194907 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.194892    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jun 15 05:49:09.205446 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.205425    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-544cd4646d-gzxjx"]
Jun 15 05:49:09.292236 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.292198    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/92ef9a64-4060-4a86-a206-d299de7e4b80-metrics-server-audit-profiles\") pod \"metrics-server-544cd4646d-gzxjx\" (UID: \"92ef9a64-4060-4a86-a206-d299de7e4b80\") " pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx"
Jun 15 05:49:09.292419 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.292245    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/92ef9a64-4060-4a86-a206-d299de7e4b80-client-ca-bundle\") pod \"metrics-server-544cd4646d-gzxjx\" (UID: \"92ef9a64-4060-4a86-a206-d299de7e4b80\") " pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx"
Jun 15 05:49:09.292419 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.292380    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/92ef9a64-4060-4a86-a206-d299de7e4b80-secret-metrics-server-tls\") pod \"metrics-server-544cd4646d-gzxjx\" (UID: \"92ef9a64-4060-4a86-a206-d299de7e4b80\") " pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx"
Jun 15 05:49:09.292553 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.292434    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/92ef9a64-4060-4a86-a206-d299de7e4b80-secret-metrics-server-client-certs\") pod \"metrics-server-544cd4646d-gzxjx\" (UID: \"92ef9a64-4060-4a86-a206-d299de7e4b80\") " pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx"
Jun 15 05:49:09.292553 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.292486    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92ef9a64-4060-4a86-a206-d299de7e4b80-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-544cd4646d-gzxjx\" (UID: \"92ef9a64-4060-4a86-a206-d299de7e4b80\") " pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx"
Jun 15 05:49:09.292553 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.292532    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/92ef9a64-4060-4a86-a206-d299de7e4b80-audit-log\") pod \"metrics-server-544cd4646d-gzxjx\" (UID: \"92ef9a64-4060-4a86-a206-d299de7e4b80\") " pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx"
Jun 15 05:49:09.292681 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.292608    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm69f\" (UniqueName: \"kubernetes.io/projected/92ef9a64-4060-4a86-a206-d299de7e4b80-kube-api-access-cm69f\") pod \"metrics-server-544cd4646d-gzxjx\" (UID: \"92ef9a64-4060-4a86-a206-d299de7e4b80\") " pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx"
Jun 15 05:49:09.299979 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.299934    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-589576dfc7-pqjxp"]
Jun 15 05:49:09.302451 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.302431    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-589576dfc7-pqjxp"
Jun 15 05:49:09.304891 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.304866    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jun 15 05:49:09.304987 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.304919    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-d7s7j\""
Jun 15 05:49:09.311387 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.311346    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-589576dfc7-pqjxp"]
Jun 15 05:49:09.393548 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.393418    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xm7d\" (UniqueName: \"kubernetes.io/projected/1ba3283e-aed0-44dd-bbd3-daca09b7cfc5-kube-api-access-5xm7d\") pod \"monitoring-plugin-589576dfc7-pqjxp\" (UID: \"1ba3283e-aed0-44dd-bbd3-daca09b7cfc5\") " pod="openshift-monitoring/monitoring-plugin-589576dfc7-pqjxp"
Jun 15 05:49:09.393548 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.393463    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1ba3283e-aed0-44dd-bbd3-daca09b7cfc5-monitoring-plugin-cert\") pod \"monitoring-plugin-589576dfc7-pqjxp\" (UID: \"1ba3283e-aed0-44dd-bbd3-daca09b7cfc5\") " pod="openshift-monitoring/monitoring-plugin-589576dfc7-pqjxp"
Jun 15 05:49:09.393548 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.393500    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/92ef9a64-4060-4a86-a206-d299de7e4b80-secret-metrics-server-tls\") pod \"metrics-server-544cd4646d-gzxjx\" (UID: \"92ef9a64-4060-4a86-a206-d299de7e4b80\") " pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx"
Jun 15 05:49:09.393833 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.393588    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/92ef9a64-4060-4a86-a206-d299de7e4b80-secret-metrics-server-client-certs\") pod \"metrics-server-544cd4646d-gzxjx\" (UID: \"92ef9a64-4060-4a86-a206-d299de7e4b80\") " pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx"
Jun 15 05:49:09.393833 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.393652    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92ef9a64-4060-4a86-a206-d299de7e4b80-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-544cd4646d-gzxjx\" (UID: \"92ef9a64-4060-4a86-a206-d299de7e4b80\") " pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx"
Jun 15 05:49:09.393833 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.393704    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/92ef9a64-4060-4a86-a206-d299de7e4b80-audit-log\") pod \"metrics-server-544cd4646d-gzxjx\" (UID: \"92ef9a64-4060-4a86-a206-d299de7e4b80\") " pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx"
Jun 15 05:49:09.393833 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.393740    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm69f\" (UniqueName: \"kubernetes.io/projected/92ef9a64-4060-4a86-a206-d299de7e4b80-kube-api-access-cm69f\") pod \"metrics-server-544cd4646d-gzxjx\" (UID: \"92ef9a64-4060-4a86-a206-d299de7e4b80\") " pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx"
Jun 15 05:49:09.393833 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.393775    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/92ef9a64-4060-4a86-a206-d299de7e4b80-metrics-server-audit-profiles\") pod \"metrics-server-544cd4646d-gzxjx\" (UID: \"92ef9a64-4060-4a86-a206-d299de7e4b80\") " pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx"
Jun 15 05:49:09.393833 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.393800    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/92ef9a64-4060-4a86-a206-d299de7e4b80-client-ca-bundle\") pod \"metrics-server-544cd4646d-gzxjx\" (UID: \"92ef9a64-4060-4a86-a206-d299de7e4b80\") " pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx"
Jun 15 05:49:09.394189 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.394151    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/92ef9a64-4060-4a86-a206-d299de7e4b80-audit-log\") pod \"metrics-server-544cd4646d-gzxjx\" (UID: \"92ef9a64-4060-4a86-a206-d299de7e4b80\") " pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx"
Jun 15 05:49:09.394752 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.394706    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/92ef9a64-4060-4a86-a206-d299de7e4b80-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-544cd4646d-gzxjx\" (UID: \"92ef9a64-4060-4a86-a206-d299de7e4b80\") " pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx"
Jun 15 05:49:09.394998 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.394939    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/92ef9a64-4060-4a86-a206-d299de7e4b80-metrics-server-audit-profiles\") pod \"metrics-server-544cd4646d-gzxjx\" (UID: \"92ef9a64-4060-4a86-a206-d299de7e4b80\") " pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx"
Jun 15 05:49:09.397050 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.397009    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/92ef9a64-4060-4a86-a206-d299de7e4b80-secret-metrics-server-client-certs\") pod \"metrics-server-544cd4646d-gzxjx\" (UID: \"92ef9a64-4060-4a86-a206-d299de7e4b80\") " pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx"
Jun 15 05:49:09.397153 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.397081    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/92ef9a64-4060-4a86-a206-d299de7e4b80-secret-metrics-server-tls\") pod \"metrics-server-544cd4646d-gzxjx\" (UID: \"92ef9a64-4060-4a86-a206-d299de7e4b80\") " pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx"
Jun 15 05:49:09.397415 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.397394    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/92ef9a64-4060-4a86-a206-d299de7e4b80-client-ca-bundle\") pod \"metrics-server-544cd4646d-gzxjx\" (UID: \"92ef9a64-4060-4a86-a206-d299de7e4b80\") " pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx"
Jun 15 05:49:09.412491 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.412441    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm69f\" (UniqueName: \"kubernetes.io/projected/92ef9a64-4060-4a86-a206-d299de7e4b80-kube-api-access-cm69f\") pod \"metrics-server-544cd4646d-gzxjx\" (UID: \"92ef9a64-4060-4a86-a206-d299de7e4b80\") " pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx"
Jun 15 05:49:09.494721 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.494628    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xm7d\" (UniqueName: \"kubernetes.io/projected/1ba3283e-aed0-44dd-bbd3-daca09b7cfc5-kube-api-access-5xm7d\") pod \"monitoring-plugin-589576dfc7-pqjxp\" (UID: \"1ba3283e-aed0-44dd-bbd3-daca09b7cfc5\") " pod="openshift-monitoring/monitoring-plugin-589576dfc7-pqjxp"
Jun 15 05:49:09.494721 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.494687    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1ba3283e-aed0-44dd-bbd3-daca09b7cfc5-monitoring-plugin-cert\") pod \"monitoring-plugin-589576dfc7-pqjxp\" (UID: \"1ba3283e-aed0-44dd-bbd3-daca09b7cfc5\") " pod="openshift-monitoring/monitoring-plugin-589576dfc7-pqjxp"
Jun 15 05:49:09.497775 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.497748    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1ba3283e-aed0-44dd-bbd3-daca09b7cfc5-monitoring-plugin-cert\") pod \"monitoring-plugin-589576dfc7-pqjxp\" (UID: \"1ba3283e-aed0-44dd-bbd3-daca09b7cfc5\") " pod="openshift-monitoring/monitoring-plugin-589576dfc7-pqjxp"
Jun 15 05:49:09.503539 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.503513    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx"
Jun 15 05:49:09.507021 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.507001    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xm7d\" (UniqueName: \"kubernetes.io/projected/1ba3283e-aed0-44dd-bbd3-daca09b7cfc5-kube-api-access-5xm7d\") pod \"monitoring-plugin-589576dfc7-pqjxp\" (UID: \"1ba3283e-aed0-44dd-bbd3-daca09b7cfc5\") " pod="openshift-monitoring/monitoring-plugin-589576dfc7-pqjxp"
Jun 15 05:49:09.618905 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.618868    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-589576dfc7-pqjxp"
Jun 15 05:49:09.649177 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.649143    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65b6bfd656-g7zrp"]
Jun 15 05:49:09.678091 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.678048    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86b56dffcb-754hc"]
Jun 15 05:49:09.685169 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.685135    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86b56dffcb-754hc"
Jun 15 05:49:09.692954 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.692926    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86b56dffcb-754hc"]
Jun 15 05:49:09.772485 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.772133    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9hzhf" event={"ID":"24202341-f48a-4a16-9619-eab661b11dc3","Type":"ContainerStarted","Data":"aac4bffb714c2cddffc6a299244af00e7350fe3399562abae496f1fbc454fdd4"}
Jun 15 05:49:09.772485 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.772201    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9hzhf" event={"ID":"24202341-f48a-4a16-9619-eab661b11dc3","Type":"ContainerStarted","Data":"928966937d6ca358fe091c1c1a91a276f0b2cc7f61f60636d9daa2eea98bc698"}
Jun 15 05:49:09.794769 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.794511    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9hzhf" podStartSLOduration=3.76598866 podStartE2EDuration="5.79449006s" totalImagesPullingTime="1.954538828s" totalInitContainerRuntime="73.962572ms" isStatefulPod=true podCreationTimestamp="2026-06-15 05:49:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:09.793814502 +0000 UTC m=+64.941994500" watchObservedRunningTime="2026-06-15 05:49:09.79449006 +0000 UTC m=+64.942670049"
Jun 15 05:49:09.798008 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.797824    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d690ffa-0e72-4a14-8b31-618a7bc880ad-console-oauth-config\") pod \"console-86b56dffcb-754hc\" (UID: \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\") " pod="openshift-console/console-86b56dffcb-754hc"
Jun 15 05:49:09.798008 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.797894    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d690ffa-0e72-4a14-8b31-618a7bc880ad-console-config\") pod \"console-86b56dffcb-754hc\" (UID: \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\") " pod="openshift-console/console-86b56dffcb-754hc"
Jun 15 05:49:09.798008 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.797943    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d690ffa-0e72-4a14-8b31-618a7bc880ad-trusted-ca-bundle\") pod \"console-86b56dffcb-754hc\" (UID: \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\") " pod="openshift-console/console-86b56dffcb-754hc"
Jun 15 05:49:09.798296 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.798051    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtr9q\" (UniqueName: \"kubernetes.io/projected/0d690ffa-0e72-4a14-8b31-618a7bc880ad-kube-api-access-dtr9q\") pod \"console-86b56dffcb-754hc\" (UID: \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\") " pod="openshift-console/console-86b56dffcb-754hc"
Jun 15 05:49:09.798296 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.798152    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d690ffa-0e72-4a14-8b31-618a7bc880ad-console-serving-cert\") pod \"console-86b56dffcb-754hc\" (UID: \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\") " pod="openshift-console/console-86b56dffcb-754hc"
Jun 15 05:49:09.798296 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.798206    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d690ffa-0e72-4a14-8b31-618a7bc880ad-service-ca\") pod \"console-86b56dffcb-754hc\" (UID: \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\") " pod="openshift-console/console-86b56dffcb-754hc"
Jun 15 05:49:09.798296 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.798236    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d690ffa-0e72-4a14-8b31-618a7bc880ad-oauth-serving-cert\") pod \"console-86b56dffcb-754hc\" (UID: \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\") " pod="openshift-console/console-86b56dffcb-754hc"
Jun 15 05:49:09.899532 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.899500    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d690ffa-0e72-4a14-8b31-618a7bc880ad-console-config\") pod \"console-86b56dffcb-754hc\" (UID: \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\") " pod="openshift-console/console-86b56dffcb-754hc"
Jun 15 05:49:09.899713 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.899663    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d690ffa-0e72-4a14-8b31-618a7bc880ad-trusted-ca-bundle\") pod \"console-86b56dffcb-754hc\" (UID: \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\") " pod="openshift-console/console-86b56dffcb-754hc"
Jun 15 05:49:09.899941 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.899911    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtr9q\" (UniqueName: \"kubernetes.io/projected/0d690ffa-0e72-4a14-8b31-618a7bc880ad-kube-api-access-dtr9q\") pod \"console-86b56dffcb-754hc\" (UID: \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\") " pod="openshift-console/console-86b56dffcb-754hc"
Jun 15 05:49:09.900061 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.900024    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d690ffa-0e72-4a14-8b31-618a7bc880ad-console-serving-cert\") pod \"console-86b56dffcb-754hc\" (UID: \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\") " pod="openshift-console/console-86b56dffcb-754hc"
Jun 15 05:49:09.900385 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.900130    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d690ffa-0e72-4a14-8b31-618a7bc880ad-service-ca\") pod \"console-86b56dffcb-754hc\" (UID: \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\") " pod="openshift-console/console-86b56dffcb-754hc"
Jun 15 05:49:09.900385 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.900162    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d690ffa-0e72-4a14-8b31-618a7bc880ad-oauth-serving-cert\") pod \"console-86b56dffcb-754hc\" (UID: \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\") " pod="openshift-console/console-86b56dffcb-754hc"
Jun 15 05:49:09.900385 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.900230    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d690ffa-0e72-4a14-8b31-618a7bc880ad-console-oauth-config\") pod \"console-86b56dffcb-754hc\" (UID: \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\") " pod="openshift-console/console-86b56dffcb-754hc"
Jun 15 05:49:09.900385 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.900256    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d690ffa-0e72-4a14-8b31-618a7bc880ad-console-config\") pod \"console-86b56dffcb-754hc\" (UID: \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\") " pod="openshift-console/console-86b56dffcb-754hc"
Jun 15 05:49:09.900651 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.900611    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d690ffa-0e72-4a14-8b31-618a7bc880ad-trusted-ca-bundle\") pod \"console-86b56dffcb-754hc\" (UID: \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\") " pod="openshift-console/console-86b56dffcb-754hc"
Jun 15 05:49:09.900769 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.900747    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d690ffa-0e72-4a14-8b31-618a7bc880ad-service-ca\") pod \"console-86b56dffcb-754hc\" (UID: \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\") " pod="openshift-console/console-86b56dffcb-754hc"
Jun 15 05:49:09.900870 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.900845    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d690ffa-0e72-4a14-8b31-618a7bc880ad-oauth-serving-cert\") pod \"console-86b56dffcb-754hc\" (UID: \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\") " pod="openshift-console/console-86b56dffcb-754hc"
Jun 15 05:49:09.902899 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.902862    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d690ffa-0e72-4a14-8b31-618a7bc880ad-console-serving-cert\") pod \"console-86b56dffcb-754hc\" (UID: \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\") " pod="openshift-console/console-86b56dffcb-754hc"
Jun 15 05:49:09.902990 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.902972    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d690ffa-0e72-4a14-8b31-618a7bc880ad-console-oauth-config\") pod \"console-86b56dffcb-754hc\" (UID: \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\") " pod="openshift-console/console-86b56dffcb-754hc"
Jun 15 05:49:09.909553 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.909534    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtr9q\" (UniqueName: \"kubernetes.io/projected/0d690ffa-0e72-4a14-8b31-618a7bc880ad-kube-api-access-dtr9q\") pod \"console-86b56dffcb-754hc\" (UID: \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\") " pod="openshift-console/console-86b56dffcb-754hc"
Jun 15 05:49:09.997597 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:09.997561    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86b56dffcb-754hc"
Jun 15 05:49:10.102412 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.102307    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c764213-65e7-4296-a8b6-023c581b36c1-original-pull-secret\") pod \"global-pull-secret-syncer-z9ws7\" (UID: \"8c764213-65e7-4296-a8b6-023c581b36c1\") " pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:49:10.105275 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.105251    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jun 15 05:49:10.113082 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.113049    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65b6bfd656-g7zrp"
Jun 15 05:49:10.116017 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.115998    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8c764213-65e7-4296-a8b6-023c581b36c1-original-pull-secret\") pod \"global-pull-secret-syncer-z9ws7\" (UID: \"8c764213-65e7-4296-a8b6-023c581b36c1\") " pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:49:10.141288 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.141254    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-z9ws7"
Jun 15 05:49:10.202893 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.202865    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57565893-704b-4974-bc5b-c8f9cb932d15-metrics-certs\") pod \"network-metrics-daemon-vs8ck\" (UID: \"57565893-704b-4974-bc5b-c8f9cb932d15\") " pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:49:10.205075 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.205051    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jun 15 05:49:10.215932 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.215905    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57565893-704b-4974-bc5b-c8f9cb932d15-metrics-certs\") pod \"network-metrics-daemon-vs8ck\" (UID: \"57565893-704b-4974-bc5b-c8f9cb932d15\") " pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:49:10.304994 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.304428    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mws52\" (UniqueName: \"kubernetes.io/projected/243c1d92-89f7-4ebb-838a-0f18bd3216d8-kube-api-access-mws52\") pod \"network-check-target-9b7jj\" (UID: \"243c1d92-89f7-4ebb-838a-0f18bd3216d8\") " pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:49:10.307274 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.307079    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jun 15 05:49:10.319837 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.319602    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jun 15 05:49:10.330782 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.330753    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mws52\" (UniqueName: \"kubernetes.io/projected/243c1d92-89f7-4ebb-838a-0f18bd3216d8-kube-api-access-mws52\") pod \"network-check-target-9b7jj\" (UID: \"243c1d92-89f7-4ebb-838a-0f18bd3216d8\") " pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:49:10.402922 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.402832    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86b56dffcb-754hc"]
Jun 15 05:49:10.408705 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:49:10.408663    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d690ffa_0e72_4a14_8b31_618a7bc880ad.slice/crio-185ef829d8664ad76e63abb2dafa8d3d4e4c880aee77623d8bbb39fd25472c3f WatchSource:0}: Error finding container 185ef829d8664ad76e63abb2dafa8d3d4e4c880aee77623d8bbb39fd25472c3f: Status 404 returned error can't find the container with id 185ef829d8664ad76e63abb2dafa8d3d4e4c880aee77623d8bbb39fd25472c3f
Jun 15 05:49:10.410693 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.410666    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-z9ws7"]
Jun 15 05:49:10.414345 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:49:10.414292    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c764213_65e7_4296_a8b6_023c581b36c1.slice/crio-612a4903f06d4d121d12ed41f33d4f31a3c8cfb6dbdb3b3fdef9b446ba647351 WatchSource:0}: Error finding container 612a4903f06d4d121d12ed41f33d4f31a3c8cfb6dbdb3b3fdef9b446ba647351: Status 404 returned error can't find the container with id 612a4903f06d4d121d12ed41f33d4f31a3c8cfb6dbdb3b3fdef9b446ba647351
Jun 15 05:49:10.424110 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.424089    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-589576dfc7-pqjxp"]
Jun 15 05:49:10.428602 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.428481    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-544cd4646d-gzxjx"]
Jun 15 05:49:10.434529 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:49:10.434212    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba3283e_aed0_44dd_bbd3_daca09b7cfc5.slice/crio-c6a3e50eb247055f2bd80cbe00cd9372ecde64759957cb0ef43c52ee997c7e06 WatchSource:0}: Error finding container c6a3e50eb247055f2bd80cbe00cd9372ecde64759957cb0ef43c52ee997c7e06: Status 404 returned error can't find the container with id c6a3e50eb247055f2bd80cbe00cd9372ecde64759957cb0ef43c52ee997c7e06
Jun 15 05:49:10.434738 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.434612    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sh6m9\""
Jun 15 05:49:10.437798 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:49:10.436951    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ef9a64_4060_4a86_a206_d299de7e4b80.slice/crio-7eaa5e2af0250ba0bd43502fd479dc767748dfb503ba92a0e0ad46939768bdf1 WatchSource:0}: Error finding container 7eaa5e2af0250ba0bd43502fd479dc767748dfb503ba92a0e0ad46939768bdf1: Status 404 returned error can't find the container with id 7eaa5e2af0250ba0bd43502fd479dc767748dfb503ba92a0e0ad46939768bdf1
Jun 15 05:49:10.443241 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.443040    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:49:10.451395 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.451070    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4mr6w\""
Jun 15 05:49:10.459866 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.459579    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vs8ck"
Jun 15 05:49:10.621058 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.620985    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-9b7jj"]
Jun 15 05:49:10.649897 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.647267    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vs8ck"]
Jun 15 05:49:10.650548 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:49:10.650514    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57565893_704b_4974_bc5b_c8f9cb932d15.slice/crio-88f9a08bd55038942e271c3f77fdbf57f9396375547cc84c6f8ffd6cba08a8a7 WatchSource:0}: Error finding container 88f9a08bd55038942e271c3f77fdbf57f9396375547cc84c6f8ffd6cba08a8a7: Status 404 returned error can't find the container with id 88f9a08bd55038942e271c3f77fdbf57f9396375547cc84c6f8ffd6cba08a8a7
Jun 15 05:49:10.777712 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.777661    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx" event={"ID":"92ef9a64-4060-4a86-a206-d299de7e4b80","Type":"ContainerStarted","Data":"7eaa5e2af0250ba0bd43502fd479dc767748dfb503ba92a0e0ad46939768bdf1"}
Jun 15 05:49:10.779237 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.779014    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-589576dfc7-pqjxp" event={"ID":"1ba3283e-aed0-44dd-bbd3-daca09b7cfc5","Type":"ContainerStarted","Data":"c6a3e50eb247055f2bd80cbe00cd9372ecde64759957cb0ef43c52ee997c7e06"}
Jun 15 05:49:10.783305 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.783279    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a5d13e-ee76-4cd0-b100-e51a330f1765","Type":"ContainerStarted","Data":"09e0774a9484eb76c4aced99b75a0a5ed432a8e474638c273b0378ac98ddb193"}
Jun 15 05:49:10.783423 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.783332    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a5d13e-ee76-4cd0-b100-e51a330f1765","Type":"ContainerStarted","Data":"86b626cb09bf3ae9244b80026402ff657d41088d8d591418145a343afe5f6225"}
Jun 15 05:49:10.783423 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.783375    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a5d13e-ee76-4cd0-b100-e51a330f1765","Type":"ContainerStarted","Data":"deb2f14ef89c342ec97f83a780c5a60cde8dca409c9966f8541d4880f8a7005d"}
Jun 15 05:49:10.783423 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.783400    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a5d13e-ee76-4cd0-b100-e51a330f1765","Type":"ContainerStarted","Data":"41b867c675e19f596b9ffd4d6bc691817b321ebab87a1c6084ead2438f45038c"}
Jun 15 05:49:10.783423 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.783423    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a5d13e-ee76-4cd0-b100-e51a330f1765","Type":"ContainerStarted","Data":"feea416259c3bc69fdbfc9c3c09434d827a430fedc5c8d95ba3f06c89d2a1a35"}
Jun 15 05:49:10.785767 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.785732    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242" event={"ID":"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef","Type":"ContainerStarted","Data":"9eec28c2add8945f7fe9bacb4d744bd9715a693884c0ebdcf9bc08672aa00b48"}
Jun 15 05:49:10.785870 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.785800    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242" event={"ID":"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef","Type":"ContainerStarted","Data":"88e9db2b0fbc54639776f9ffe012714f910060d83cb4e0b5e03d806e59678626"}
Jun 15 05:49:10.785870 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.785841    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242" event={"ID":"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef","Type":"ContainerStarted","Data":"fb63c3591e9cf834dcddd1c202226158c69d14ad4ef454b4191f70b69536b171"}
Jun 15 05:49:10.787127 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.787100    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vs8ck" event={"ID":"57565893-704b-4974-bc5b-c8f9cb932d15","Type":"ContainerStarted","Data":"88f9a08bd55038942e271c3f77fdbf57f9396375547cc84c6f8ffd6cba08a8a7"}
Jun 15 05:49:10.788540 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.788493    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9b7jj" event={"ID":"243c1d92-89f7-4ebb-838a-0f18bd3216d8","Type":"ContainerStarted","Data":"2a3801ac2610cec5834b9b0de57af94601caefe140adffdcca282bbd28395d27"}
Jun 15 05:49:10.790350 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.790307    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86b56dffcb-754hc" event={"ID":"0d690ffa-0e72-4a14-8b31-618a7bc880ad","Type":"ContainerStarted","Data":"f7f05636a9c9faa3d5aad5bf04ce373d1f9f3ec16c3980d240644775ac9a3694"}
Jun 15 05:49:10.790455 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.790394    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86b56dffcb-754hc" event={"ID":"0d690ffa-0e72-4a14-8b31-618a7bc880ad","Type":"ContainerStarted","Data":"185ef829d8664ad76e63abb2dafa8d3d4e4c880aee77623d8bbb39fd25472c3f"}
Jun 15 05:49:10.791705 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.791684    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z9ws7" event={"ID":"8c764213-65e7-4296-a8b6-023c581b36c1","Type":"ContainerStarted","Data":"612a4903f06d4d121d12ed41f33d4f31a3c8cfb6dbdb3b3fdef9b446ba647351"}
Jun 15 05:49:10.814453 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.814395    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86b56dffcb-754hc" podStartSLOduration=1.814373703 podStartE2EDuration="1.814373703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:10.812764918 +0000 UTC m=+65.960944915" watchObservedRunningTime="2026-06-15 05:49:10.814373703 +0000 UTC m=+65.962553697"
Jun 15 05:49:10.869062 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.868984    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jun 15 05:49:10.874270 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.874245    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:10.877395 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.877018    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jun 15 05:49:10.877395 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.877118    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jun 15 05:49:10.881706 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.878562    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jun 15 05:49:10.881706 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.878649    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jun 15 05:49:10.883959 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.883935    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jun 15 05:49:10.884113 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.884091    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jun 15 05:49:10.884186 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.884003    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vhmkr\""
Jun 15 05:49:10.884186 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.884177    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jun 15 05:49:10.884282 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.884222    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jun 15 05:49:10.884352 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.884041    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jun 15 05:49:10.884540 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.883956    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jun 15 05:49:10.884649 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.884469    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dhj5i5bidigv1\""
Jun 15 05:49:10.884811 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.884574    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jun 15 05:49:10.884910 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.884872    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jun 15 05:49:10.889757 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.889636    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jun 15 05:49:10.893596 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.893397    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jun 15 05:49:10.894727 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:10.894707    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jun 15 05:49:11.012180 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.011774    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.012180 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.011829    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.012180 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.011853    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.012180 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.011891    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.012180 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.011917    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-web-config\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.012180 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.011943    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.012180 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.011972    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.012180 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.012000    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.012180 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.012023    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsfmk\" (UniqueName: \"kubernetes.io/projected/e0dabd34-b321-41af-8732-7c7b5b8eac93-kube-api-access-vsfmk\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.012180 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.012056    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.012180 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.012094    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.012180 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.012118    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.012180 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.012148    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-config\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.012180 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.012172    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0dabd34-b321-41af-8732-7c7b5b8eac93-config-out\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.012180 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.012195    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0dabd34-b321-41af-8732-7c7b5b8eac93-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.014150 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.012222    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.014150 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.012279    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.014150 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.012304    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.014150 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.012353    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.014150 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.012400    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.115538 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.113796    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.115538 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.113850    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.115538 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.113913    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.115538 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.113943    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.115538 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.114033    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.115538 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.114120    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.115538 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.114148    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-web-config\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.115538 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.114175    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.115538 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.114205    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.115538 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.114232    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.115538 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.114259    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsfmk\" (UniqueName: \"kubernetes.io/projected/e0dabd34-b321-41af-8732-7c7b5b8eac93-kube-api-access-vsfmk\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.115538 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.114296    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.115538 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.114338    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.115538 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.114380    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.115538 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.114385    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.115538 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.114445    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-config\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.115538 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.114471    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0dabd34-b321-41af-8732-7c7b5b8eac93-config-out\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.116466 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.114498    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0dabd34-b321-41af-8732-7c7b5b8eac93-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.116466 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.114526    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.116466 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.114557    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.116466 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.114585    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.116466 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.114858    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.116466 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.115260    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.116466 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.115386    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.117661 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.116919    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.117661 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.117386    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.120677 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.120619    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.121830 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.121259    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.121830 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.121504    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.125330 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.125309    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsfmk\" (UniqueName: \"kubernetes.io/projected/e0dabd34-b321-41af-8732-7c7b5b8eac93-kube-api-access-vsfmk\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.129940 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.129912    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.132706 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.132660    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0dabd34-b321-41af-8732-7c7b5b8eac93-config-out\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.134186 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.134146    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-config\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.134749 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.134471    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0dabd34-b321-41af-8732-7c7b5b8eac93-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.134749 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.134713    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.135901 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.135873    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.142107 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.142054    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-web-config\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.142519 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.142496    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.142725 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.142695    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.145410 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.145386    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.197153 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.196763    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:11.468822 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.468714    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jun 15 05:49:11.479816 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:49:11.479764    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0dabd34_b321_41af_8732_7c7b5b8eac93.slice/crio-8630064b0605ce964b2b0b0ddd23abc0040dbd42285635d643cad419560782ac WatchSource:0}: Error finding container 8630064b0605ce964b2b0b0ddd23abc0040dbd42285635d643cad419560782ac: Status 404 returned error can't find the container with id 8630064b0605ce964b2b0b0ddd23abc0040dbd42285635d643cad419560782ac
Jun 15 05:49:11.721034 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.720080    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-q6c87"
Jun 15 05:49:11.812293 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.811460    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0dabd34-b321-41af-8732-7c7b5b8eac93","Type":"ContainerStarted","Data":"258a8c4669eb7412843b979eea3eec5eca6d71bcfb4fb2d8c1872d4b5082271b"}
Jun 15 05:49:11.812293 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:11.811611    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0dabd34-b321-41af-8732-7c7b5b8eac93","Type":"ContainerStarted","Data":"8630064b0605ce964b2b0b0ddd23abc0040dbd42285635d643cad419560782ac"}
Jun 15 05:49:12.820129 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:12.820040    2572 generic.go:356] "Generic (PLEG): container finished" podID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerID="258a8c4669eb7412843b979eea3eec5eca6d71bcfb4fb2d8c1872d4b5082271b" exitCode=0
Jun 15 05:49:12.820607 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:12.820146    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0dabd34-b321-41af-8732-7c7b5b8eac93","Type":"ContainerDied","Data":"258a8c4669eb7412843b979eea3eec5eca6d71bcfb4fb2d8c1872d4b5082271b"}
Jun 15 05:49:17.861497 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:17.860924    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a5d13e-ee76-4cd0-b100-e51a330f1765","Type":"ContainerStarted","Data":"6ecdf406a1d8b8e27c52bc2cfb9e212e43baaafdf4128254cf22805977c1e199"}
Jun 15 05:49:17.865095 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:17.865069    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242" event={"ID":"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef","Type":"ContainerStarted","Data":"5d43e52d7d1452b61d25dd36e33fe2f8134b54d439ae20bb3776cf9cfed51c29"}
Jun 15 05:49:17.865193 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:17.865146    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242" event={"ID":"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef","Type":"ContainerStarted","Data":"b1e893daba22f08fcd2b76bf90190dea7e81aefe8fbd8483747d993e719702a2"}
Jun 15 05:49:17.873423 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:17.872924    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vs8ck" event={"ID":"57565893-704b-4974-bc5b-c8f9cb932d15","Type":"ContainerStarted","Data":"4f0931d43f4f7b65c4e8db2a2aa0c1a3e50f391501a71b41d9b07befe2a938bd"}
Jun 15 05:49:17.873423 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:17.872984    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vs8ck" event={"ID":"57565893-704b-4974-bc5b-c8f9cb932d15","Type":"ContainerStarted","Data":"bec3f7aaa7986efa361cb4c640cfa686281a871e3b2b56eec7d51c618bba3fe4"}
Jun 15 05:49:17.875781 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:17.875750    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-9b7jj" event={"ID":"243c1d92-89f7-4ebb-838a-0f18bd3216d8","Type":"ContainerStarted","Data":"95237c6a2abc70f38036b381378f8bfd53419daaad41355b581331f8e6e59471"}
Jun 15 05:49:17.876302 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:17.876282    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:49:17.881104 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:17.880956    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0dabd34-b321-41af-8732-7c7b5b8eac93","Type":"ContainerStarted","Data":"80025a1efc19d179e7472309d0282d6a382fc1f37da5ea681213a950cddfc0b9"}
Jun 15 05:49:17.881104 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:17.881032    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0dabd34-b321-41af-8732-7c7b5b8eac93","Type":"ContainerStarted","Data":"70eaf93693bdf8662e2fcb41a1e19df19e8e8af5f64b047edc3d5c24cfb5488b"}
Jun 15 05:49:17.883211 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:17.883160    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-z9ws7" event={"ID":"8c764213-65e7-4296-a8b6-023c581b36c1","Type":"ContainerStarted","Data":"90b98959ab064dde877e60fd3b307debbaa9d0974387ab4c506043c8bd24026e"}
Jun 15 05:49:17.885494 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:17.885473    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx" event={"ID":"92ef9a64-4060-4a86-a206-d299de7e4b80","Type":"ContainerStarted","Data":"efbf7e21ba114cee050a7acca0a2b60d7efbb47d5f19e3eea713c5e601ce4fe7"}
Jun 15 05:49:17.889164 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:17.887741    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-589576dfc7-pqjxp" event={"ID":"1ba3283e-aed0-44dd-bbd3-daca09b7cfc5","Type":"ContainerStarted","Data":"a7ca92157b5b46467c94478b331dce8eeb324cd5462695190423586821e40ff3"}
Jun 15 05:49:17.889164 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:17.888522    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-589576dfc7-pqjxp"
Jun 15 05:49:17.898270 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:17.898210    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-589576dfc7-pqjxp"
Jun 15 05:49:17.905642 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:17.905593    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.568332575 podStartE2EDuration="12.905055826s" totalImagesPullingTime="8.318711882s" totalInitContainerRuntime="18.011369ms" isStatefulPod=false podCreationTimestamp="2026-06-15 05:49:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:17.904505284 +0000 UTC m=+73.052685294" watchObservedRunningTime="2026-06-15 05:49:17.905055826 +0000 UTC m=+73.053235834"
Jun 15 05:49:17.928631 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:17.924843    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-9b7jj" podStartSLOduration=66.046266339 podStartE2EDuration="1m12.924805444s" totalImagesPullingTime="6.878539105s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:48:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:17.924663235 +0000 UTC m=+73.072843233" watchObservedRunningTime="2026-06-15 05:49:17.924805444 +0000 UTC m=+73.072985444"
Jun 15 05:49:17.967737 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:17.967002    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-589576dfc7-pqjxp" podStartSLOduration=1.896861884 podStartE2EDuration="8.966979227s" totalImagesPullingTime="7.070117343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:49:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:17.956077161 +0000 UTC m=+73.104257160" watchObservedRunningTime="2026-06-15 05:49:17.966979227 +0000 UTC m=+73.115159225"
Jun 15 05:49:17.980953 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:17.979881    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-z9ws7" podStartSLOduration=65.89101514 podStartE2EDuration="1m12.979858553s" totalImagesPullingTime="7.088843413s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 05:48:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:17.975705473 +0000 UTC m=+73.123885471" watchObservedRunningTime="2026-06-15 05:49:17.979858553 +0000 UTC m=+73.128038562"
Jun 15 05:49:18.021465 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:18.021071    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vs8ck" podStartSLOduration=66.167287771 podStartE2EDuration="1m13.021057011s" totalImagesPullingTime="6.85376924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:48:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:17.996270276 +0000 UTC m=+73.144450274" watchObservedRunningTime="2026-06-15 05:49:18.021057011 +0000 UTC m=+73.169237011"
Jun 15 05:49:18.021465 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:18.021324    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx" podStartSLOduration=1.9539643089999998 podStartE2EDuration="9.021315066s" totalImagesPullingTime="7.067350757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:49:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:18.02094837 +0000 UTC m=+73.169128368" watchObservedRunningTime="2026-06-15 05:49:18.021315066 +0000 UTC m=+73.169495063"
Jun 15 05:49:18.895860 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:18.895820    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0dabd34-b321-41af-8732-7c7b5b8eac93","Type":"ContainerStarted","Data":"be2f002fa7d70c3868eb5eab904311daed52c62bd1030e1df483f16d34584f44"}
Jun 15 05:49:18.896340 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:18.895944    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0dabd34-b321-41af-8732-7c7b5b8eac93","Type":"ContainerStarted","Data":"0419e6565f157615b49a2a9f47d3fa15efc95e5fd4339db5edd64b29e23e860b"}
Jun 15 05:49:18.896340 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:18.895989    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0dabd34-b321-41af-8732-7c7b5b8eac93","Type":"ContainerStarted","Data":"32af28c467b33602f23c086efa5fcdc1324825499a80fe80bff358f6b665b258"}
Jun 15 05:49:18.896340 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:18.896037    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0dabd34-b321-41af-8732-7c7b5b8eac93","Type":"ContainerStarted","Data":"50d6795699cf4515fdb42080aafcd623fa367ac9155e029e8044ade0c5769f28"}
Jun 15 05:49:18.898711 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:18.898680    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242" event={"ID":"d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef","Type":"ContainerStarted","Data":"eb72d292196cb879348633e62b4da52b7080b3ac980f36bf4d930270557d60a9"}
Jun 15 05:49:18.899236 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:18.899214    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242"
Jun 15 05:49:18.904421 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:18.904403    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242"
Jun 15 05:49:18.928549 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:18.928497    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=4.157112367 podStartE2EDuration="8.928482524s" totalImagesPullingTime="4.750721482s" totalInitContainerRuntime="20.648675ms" isStatefulPod=false podCreationTimestamp="2026-06-15 05:49:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:18.925637919 +0000 UTC m=+74.073817916" watchObservedRunningTime="2026-06-15 05:49:18.928482524 +0000 UTC m=+74.076662575"
Jun 15 05:49:18.949787 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:18.949736    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-569bcc98cc-w6242" podStartSLOduration=2.796576968 podStartE2EDuration="11.949719436s" totalImagesPullingTime="9.153142468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:49:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:49:18.948665388 +0000 UTC m=+74.096845388" watchObservedRunningTime="2026-06-15 05:49:18.949719436 +0000 UTC m=+74.097899432"
Jun 15 05:49:19.998125 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:19.998071    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86b56dffcb-754hc"
Jun 15 05:49:19.998517 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:19.998145    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86b56dffcb-754hc"
Jun 15 05:49:20.002966 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:20.002943    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86b56dffcb-754hc"
Jun 15 05:49:20.913126 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:20.913100    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86b56dffcb-754hc"
Jun 15 05:49:21.198247 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:21.198160    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:49:29.504449 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:29.504412    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx"
Jun 15 05:49:29.504898 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:29.504492    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx"
Jun 15 05:49:34.675605 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:34.675566    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-65b6bfd656-g7zrp" podUID="77cb750c-2c2f-41b9-ada6-7021f53ea271" containerName="console" containerID="cri-o://fcfcb3b9d2cc725fa3840803da87a3af8171ac7641ba6503576e00f41d81d4bf" gracePeriod=15
Jun 15 05:49:34.975242 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:34.975162    2572 generic.go:356] "Generic (PLEG): container finished" podID="77cb750c-2c2f-41b9-ada6-7021f53ea271" containerID="fcfcb3b9d2cc725fa3840803da87a3af8171ac7641ba6503576e00f41d81d4bf" exitCode=0
Jun 15 05:49:34.975397 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:34.975249    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65b6bfd656-g7zrp" event={"ID":"77cb750c-2c2f-41b9-ada6-7021f53ea271","Type":"ContainerDied","Data":"fcfcb3b9d2cc725fa3840803da87a3af8171ac7641ba6503576e00f41d81d4bf"}
Jun 15 05:49:35.463588 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:35.463566    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65b6bfd656-g7zrp"
Jun 15 05:49:35.568296 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:35.568207    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/77cb750c-2c2f-41b9-ada6-7021f53ea271-console-oauth-config\") pod \"77cb750c-2c2f-41b9-ada6-7021f53ea271\" (UID: \"77cb750c-2c2f-41b9-ada6-7021f53ea271\") "
Jun 15 05:49:35.568296 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:35.568296    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/77cb750c-2c2f-41b9-ada6-7021f53ea271-oauth-serving-cert\") pod \"77cb750c-2c2f-41b9-ada6-7021f53ea271\" (UID: \"77cb750c-2c2f-41b9-ada6-7021f53ea271\") "
Jun 15 05:49:35.568553 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:35.568389    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vx8jw\" (UniqueName: \"kubernetes.io/projected/77cb750c-2c2f-41b9-ada6-7021f53ea271-kube-api-access-vx8jw\") pod \"77cb750c-2c2f-41b9-ada6-7021f53ea271\" (UID: \"77cb750c-2c2f-41b9-ada6-7021f53ea271\") "
Jun 15 05:49:35.568553 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:35.568421    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/77cb750c-2c2f-41b9-ada6-7021f53ea271-console-serving-cert\") pod \"77cb750c-2c2f-41b9-ada6-7021f53ea271\" (UID: \"77cb750c-2c2f-41b9-ada6-7021f53ea271\") "
Jun 15 05:49:35.568553 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:35.568455    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77cb750c-2c2f-41b9-ada6-7021f53ea271-trusted-ca-bundle\") pod \"77cb750c-2c2f-41b9-ada6-7021f53ea271\" (UID: \"77cb750c-2c2f-41b9-ada6-7021f53ea271\") "
Jun 15 05:49:35.568553 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:35.568477    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/77cb750c-2c2f-41b9-ada6-7021f53ea271-console-config\") pod \"77cb750c-2c2f-41b9-ada6-7021f53ea271\" (UID: \"77cb750c-2c2f-41b9-ada6-7021f53ea271\") "
Jun 15 05:49:35.568553 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:35.568500    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/77cb750c-2c2f-41b9-ada6-7021f53ea271-service-ca\") pod \"77cb750c-2c2f-41b9-ada6-7021f53ea271\" (UID: \"77cb750c-2c2f-41b9-ada6-7021f53ea271\") "
Jun 15 05:49:35.568936 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:35.568901    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77cb750c-2c2f-41b9-ada6-7021f53ea271-console-config" (OuterVolumeSpecName: "console-config") pod "77cb750c-2c2f-41b9-ada6-7021f53ea271" (UID: "77cb750c-2c2f-41b9-ada6-7021f53ea271"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:49:35.569016 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:35.568945    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77cb750c-2c2f-41b9-ada6-7021f53ea271-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "77cb750c-2c2f-41b9-ada6-7021f53ea271" (UID: "77cb750c-2c2f-41b9-ada6-7021f53ea271"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:49:35.569016 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:35.568968    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77cb750c-2c2f-41b9-ada6-7021f53ea271-service-ca" (OuterVolumeSpecName: "service-ca") pod "77cb750c-2c2f-41b9-ada6-7021f53ea271" (UID: "77cb750c-2c2f-41b9-ada6-7021f53ea271"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:49:35.569121 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:35.569014    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77cb750c-2c2f-41b9-ada6-7021f53ea271-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "77cb750c-2c2f-41b9-ada6-7021f53ea271" (UID: "77cb750c-2c2f-41b9-ada6-7021f53ea271"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:49:35.570699 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:35.570674    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77cb750c-2c2f-41b9-ada6-7021f53ea271-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "77cb750c-2c2f-41b9-ada6-7021f53ea271" (UID: "77cb750c-2c2f-41b9-ada6-7021f53ea271"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:49:35.570699 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:35.570684    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77cb750c-2c2f-41b9-ada6-7021f53ea271-kube-api-access-vx8jw" (OuterVolumeSpecName: "kube-api-access-vx8jw") pod "77cb750c-2c2f-41b9-ada6-7021f53ea271" (UID: "77cb750c-2c2f-41b9-ada6-7021f53ea271"). InnerVolumeSpecName "kube-api-access-vx8jw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 05:49:35.570845 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:35.570770    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77cb750c-2c2f-41b9-ada6-7021f53ea271-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "77cb750c-2c2f-41b9-ada6-7021f53ea271" (UID: "77cb750c-2c2f-41b9-ada6-7021f53ea271"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:49:35.669426 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:35.669397    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx8jw\" (UniqueName: \"kubernetes.io/projected/77cb750c-2c2f-41b9-ada6-7021f53ea271-kube-api-access-vx8jw\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:49:35.669426 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:35.669423    2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/77cb750c-2c2f-41b9-ada6-7021f53ea271-console-serving-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:49:35.669426 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:35.669432    2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77cb750c-2c2f-41b9-ada6-7021f53ea271-trusted-ca-bundle\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:49:35.669641 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:35.669443    2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/77cb750c-2c2f-41b9-ada6-7021f53ea271-console-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:49:35.669641 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:35.669452    2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/77cb750c-2c2f-41b9-ada6-7021f53ea271-service-ca\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:49:35.669641 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:35.669459    2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/77cb750c-2c2f-41b9-ada6-7021f53ea271-console-oauth-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:49:35.669641 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:35.669467    2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/77cb750c-2c2f-41b9-ada6-7021f53ea271-oauth-serving-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:49:35.980553 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:35.980524    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65b6bfd656-g7zrp"
Jun 15 05:49:35.980553 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:35.980533    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65b6bfd656-g7zrp" event={"ID":"77cb750c-2c2f-41b9-ada6-7021f53ea271","Type":"ContainerDied","Data":"de1126c4839f05229fdc5d4e841d684c7f9d2f247f47f72cc61a56e8a43fc147"}
Jun 15 05:49:35.981124 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:35.980610    2572 scope.go:117] "RemoveContainer" containerID="fcfcb3b9d2cc725fa3840803da87a3af8171ac7641ba6503576e00f41d81d4bf"
Jun 15 05:49:36.002617 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:36.002593    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65b6bfd656-g7zrp"]
Jun 15 05:49:36.007173 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:36.007152    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65b6bfd656-g7zrp"]
Jun 15 05:49:37.323231 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:37.323200    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77cb750c-2c2f-41b9-ada6-7021f53ea271" path="/var/lib/kubelet/pods/77cb750c-2c2f-41b9-ada6-7021f53ea271/volumes"
Jun 15 05:49:49.509402 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:49.509368    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx"
Jun 15 05:49:49.513173 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:49.513144    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-544cd4646d-gzxjx"
Jun 15 05:49:49.907480 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:49:49.907454    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-9b7jj"
Jun 15 05:50:11.197447 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:11.197415    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:11.216660 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:11.216631    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:12.139131 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:12.139096    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:24.997790 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:24.997754    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jun 15 05:50:24.998377 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:24.998044    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerName="alertmanager" containerID="cri-o://feea416259c3bc69fdbfc9c3c09434d827a430fedc5c8d95ba3f06c89d2a1a35" gracePeriod=120
Jun 15 05:50:24.998377 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:24.998110    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerName="kube-rbac-proxy-metric" containerID="cri-o://09e0774a9484eb76c4aced99b75a0a5ed432a8e474638c273b0378ac98ddb193" gracePeriod=120
Jun 15 05:50:24.998377 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:24.998165    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerName="prom-label-proxy" containerID="cri-o://6ecdf406a1d8b8e27c52bc2cfb9e212e43baaafdf4128254cf22805977c1e199" gracePeriod=120
Jun 15 05:50:24.998377 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:24.998180    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerName="kube-rbac-proxy" containerID="cri-o://86b626cb09bf3ae9244b80026402ff657d41088d8d591418145a343afe5f6225" gracePeriod=120
Jun 15 05:50:24.998377 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:24.998129    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerName="kube-rbac-proxy-web" containerID="cri-o://deb2f14ef89c342ec97f83a780c5a60cde8dca409c9966f8541d4880f8a7005d" gracePeriod=120
Jun 15 05:50:24.998377 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:24.998192    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerName="config-reloader" containerID="cri-o://41b867c675e19f596b9ffd4d6bc691817b321ebab87a1c6084ead2438f45038c" gracePeriod=120
Jun 15 05:50:25.180551 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:25.180518    2572 generic.go:356] "Generic (PLEG): container finished" podID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerID="6ecdf406a1d8b8e27c52bc2cfb9e212e43baaafdf4128254cf22805977c1e199" exitCode=0
Jun 15 05:50:25.180551 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:25.180543    2572 generic.go:356] "Generic (PLEG): container finished" podID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerID="09e0774a9484eb76c4aced99b75a0a5ed432a8e474638c273b0378ac98ddb193" exitCode=0
Jun 15 05:50:25.180551 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:25.180550    2572 generic.go:356] "Generic (PLEG): container finished" podID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerID="86b626cb09bf3ae9244b80026402ff657d41088d8d591418145a343afe5f6225" exitCode=0
Jun 15 05:50:25.180551 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:25.180555    2572 generic.go:356] "Generic (PLEG): container finished" podID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerID="41b867c675e19f596b9ffd4d6bc691817b321ebab87a1c6084ead2438f45038c" exitCode=0
Jun 15 05:50:25.180795 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:25.180561    2572 generic.go:356] "Generic (PLEG): container finished" podID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerID="feea416259c3bc69fdbfc9c3c09434d827a430fedc5c8d95ba3f06c89d2a1a35" exitCode=0
Jun 15 05:50:25.180795 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:25.180591    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a5d13e-ee76-4cd0-b100-e51a330f1765","Type":"ContainerDied","Data":"6ecdf406a1d8b8e27c52bc2cfb9e212e43baaafdf4128254cf22805977c1e199"}
Jun 15 05:50:25.180795 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:25.180654    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a5d13e-ee76-4cd0-b100-e51a330f1765","Type":"ContainerDied","Data":"09e0774a9484eb76c4aced99b75a0a5ed432a8e474638c273b0378ac98ddb193"}
Jun 15 05:50:25.180795 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:25.180680    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a5d13e-ee76-4cd0-b100-e51a330f1765","Type":"ContainerDied","Data":"86b626cb09bf3ae9244b80026402ff657d41088d8d591418145a343afe5f6225"}
Jun 15 05:50:25.180795 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:25.180706    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a5d13e-ee76-4cd0-b100-e51a330f1765","Type":"ContainerDied","Data":"41b867c675e19f596b9ffd4d6bc691817b321ebab87a1c6084ead2438f45038c"}
Jun 15 05:50:25.180795 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:25.180730    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a5d13e-ee76-4cd0-b100-e51a330f1765","Type":"ContainerDied","Data":"feea416259c3bc69fdbfc9c3c09434d827a430fedc5c8d95ba3f06c89d2a1a35"}
Jun 15 05:50:26.189147 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.189118    2572 generic.go:356] "Generic (PLEG): container finished" podID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerID="deb2f14ef89c342ec97f83a780c5a60cde8dca409c9966f8541d4880f8a7005d" exitCode=0
Jun 15 05:50:26.189504 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.189165    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a5d13e-ee76-4cd0-b100-e51a330f1765","Type":"ContainerDied","Data":"deb2f14ef89c342ec97f83a780c5a60cde8dca409c9966f8541d4880f8a7005d"}
Jun 15 05:50:26.237200 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.237179    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:26.325404 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.325307    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-config-volume\") pod \"08a5d13e-ee76-4cd0-b100-e51a330f1765\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") "
Jun 15 05:50:26.325404 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.325340    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-cluster-tls-config\") pod \"08a5d13e-ee76-4cd0-b100-e51a330f1765\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") "
Jun 15 05:50:26.325404 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.325382    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-secret-alertmanager-kube-rbac-proxy-web\") pod \"08a5d13e-ee76-4cd0-b100-e51a330f1765\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") "
Jun 15 05:50:26.325648 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.325528    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-secret-alertmanager-kube-rbac-proxy\") pod \"08a5d13e-ee76-4cd0-b100-e51a330f1765\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") "
Jun 15 05:50:26.325648 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.325576    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfzxh\" (UniqueName: \"kubernetes.io/projected/08a5d13e-ee76-4cd0-b100-e51a330f1765-kube-api-access-rfzxh\") pod \"08a5d13e-ee76-4cd0-b100-e51a330f1765\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") "
Jun 15 05:50:26.325648 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.325617    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-secret-alertmanager-kube-rbac-proxy-metric\") pod \"08a5d13e-ee76-4cd0-b100-e51a330f1765\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") "
Jun 15 05:50:26.325809 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.325654    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/08a5d13e-ee76-4cd0-b100-e51a330f1765-alertmanager-main-db\") pod \"08a5d13e-ee76-4cd0-b100-e51a330f1765\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") "
Jun 15 05:50:26.325809 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.325695    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08a5d13e-ee76-4cd0-b100-e51a330f1765-alertmanager-trusted-ca-bundle\") pod \"08a5d13e-ee76-4cd0-b100-e51a330f1765\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") "
Jun 15 05:50:26.325809 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.325723    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-web-config\") pod \"08a5d13e-ee76-4cd0-b100-e51a330f1765\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") "
Jun 15 05:50:26.326309 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.326127    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a5d13e-ee76-4cd0-b100-e51a330f1765-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "08a5d13e-ee76-4cd0-b100-e51a330f1765" (UID: "08a5d13e-ee76-4cd0-b100-e51a330f1765"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 05:50:26.326901 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.326873    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08a5d13e-ee76-4cd0-b100-e51a330f1765-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "08a5d13e-ee76-4cd0-b100-e51a330f1765" (UID: "08a5d13e-ee76-4cd0-b100-e51a330f1765"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:50:26.327062 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.327046    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/08a5d13e-ee76-4cd0-b100-e51a330f1765-tls-assets\") pod \"08a5d13e-ee76-4cd0-b100-e51a330f1765\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") "
Jun 15 05:50:26.327193 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.327178    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-secret-alertmanager-main-tls\") pod \"08a5d13e-ee76-4cd0-b100-e51a330f1765\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") "
Jun 15 05:50:26.327331 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.327316    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08a5d13e-ee76-4cd0-b100-e51a330f1765-metrics-client-ca\") pod \"08a5d13e-ee76-4cd0-b100-e51a330f1765\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") "
Jun 15 05:50:26.327470 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.327454    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/08a5d13e-ee76-4cd0-b100-e51a330f1765-config-out\") pod \"08a5d13e-ee76-4cd0-b100-e51a330f1765\" (UID: \"08a5d13e-ee76-4cd0-b100-e51a330f1765\") "
Jun 15 05:50:26.327853 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.327833    2572 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/08a5d13e-ee76-4cd0-b100-e51a330f1765-alertmanager-main-db\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:26.327939 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.327856    2572 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/08a5d13e-ee76-4cd0-b100-e51a330f1765-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:26.328510 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.328445    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-config-volume" (OuterVolumeSpecName: "config-volume") pod "08a5d13e-ee76-4cd0-b100-e51a330f1765" (UID: "08a5d13e-ee76-4cd0-b100-e51a330f1765"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:50:26.329054 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.329024    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "08a5d13e-ee76-4cd0-b100-e51a330f1765" (UID: "08a5d13e-ee76-4cd0-b100-e51a330f1765"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:50:26.329682 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.329582    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "08a5d13e-ee76-4cd0-b100-e51a330f1765" (UID: "08a5d13e-ee76-4cd0-b100-e51a330f1765"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:50:26.329682 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.329614    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "08a5d13e-ee76-4cd0-b100-e51a330f1765" (UID: "08a5d13e-ee76-4cd0-b100-e51a330f1765"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:50:26.329989 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.329957    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08a5d13e-ee76-4cd0-b100-e51a330f1765-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "08a5d13e-ee76-4cd0-b100-e51a330f1765" (UID: "08a5d13e-ee76-4cd0-b100-e51a330f1765"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:50:26.330129 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.330106    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a5d13e-ee76-4cd0-b100-e51a330f1765-kube-api-access-rfzxh" (OuterVolumeSpecName: "kube-api-access-rfzxh") pod "08a5d13e-ee76-4cd0-b100-e51a330f1765" (UID: "08a5d13e-ee76-4cd0-b100-e51a330f1765"). InnerVolumeSpecName "kube-api-access-rfzxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 05:50:26.330785 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.330723    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08a5d13e-ee76-4cd0-b100-e51a330f1765-config-out" (OuterVolumeSpecName: "config-out") pod "08a5d13e-ee76-4cd0-b100-e51a330f1765" (UID: "08a5d13e-ee76-4cd0-b100-e51a330f1765"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 05:50:26.330785 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.330748    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "08a5d13e-ee76-4cd0-b100-e51a330f1765" (UID: "08a5d13e-ee76-4cd0-b100-e51a330f1765"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:50:26.331405 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.331376    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08a5d13e-ee76-4cd0-b100-e51a330f1765-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "08a5d13e-ee76-4cd0-b100-e51a330f1765" (UID: "08a5d13e-ee76-4cd0-b100-e51a330f1765"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 05:50:26.333939 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.333914    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "08a5d13e-ee76-4cd0-b100-e51a330f1765" (UID: "08a5d13e-ee76-4cd0-b100-e51a330f1765"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:50:26.340516 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.340482    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-web-config" (OuterVolumeSpecName: "web-config") pod "08a5d13e-ee76-4cd0-b100-e51a330f1765" (UID: "08a5d13e-ee76-4cd0-b100-e51a330f1765"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:50:26.429180 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.429041    2572 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08a5d13e-ee76-4cd0-b100-e51a330f1765-metrics-client-ca\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:26.429180 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.429075    2572 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/08a5d13e-ee76-4cd0-b100-e51a330f1765-config-out\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:26.429180 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.429089    2572 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-config-volume\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:26.429180 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.429103    2572 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-cluster-tls-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:26.429180 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.429119    2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:26.429180 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.429132    2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:26.429180 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.429161    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfzxh\" (UniqueName: \"kubernetes.io/projected/08a5d13e-ee76-4cd0-b100-e51a330f1765-kube-api-access-rfzxh\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:26.429180 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.429188    2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:26.429566 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.429203    2572 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-web-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:26.429566 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.429214    2572 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/08a5d13e-ee76-4cd0-b100-e51a330f1765-tls-assets\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:26.429566 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:26.429226    2572 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/08a5d13e-ee76-4cd0-b100-e51a330f1765-secret-alertmanager-main-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:27.196316 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.196285    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"08a5d13e-ee76-4cd0-b100-e51a330f1765","Type":"ContainerDied","Data":"f4f15bf77a99aea77517191c5c7c2f3e0f428fc2aa1417140b4921b2af839c9f"}
Jun 15 05:50:27.196316 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.196311    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.196759 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.196374    2572 scope.go:117] "RemoveContainer" containerID="6ecdf406a1d8b8e27c52bc2cfb9e212e43baaafdf4128254cf22805977c1e199"
Jun 15 05:50:27.204652 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.204633    2572 scope.go:117] "RemoveContainer" containerID="09e0774a9484eb76c4aced99b75a0a5ed432a8e474638c273b0378ac98ddb193"
Jun 15 05:50:27.211568 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.211552    2572 scope.go:117] "RemoveContainer" containerID="86b626cb09bf3ae9244b80026402ff657d41088d8d591418145a343afe5f6225"
Jun 15 05:50:27.218123 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.218105    2572 scope.go:117] "RemoveContainer" containerID="deb2f14ef89c342ec97f83a780c5a60cde8dca409c9966f8541d4880f8a7005d"
Jun 15 05:50:27.225192 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.225012    2572 scope.go:117] "RemoveContainer" containerID="41b867c675e19f596b9ffd4d6bc691817b321ebab87a1c6084ead2438f45038c"
Jun 15 05:50:27.227160 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.227139    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jun 15 05:50:27.232990 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.232973    2572 scope.go:117] "RemoveContainer" containerID="feea416259c3bc69fdbfc9c3c09434d827a430fedc5c8d95ba3f06c89d2a1a35"
Jun 15 05:50:27.233201 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.233180    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jun 15 05:50:27.239973 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.239961    2572 scope.go:117] "RemoveContainer" containerID="2a0e9a629ef24ef07c680061486c63aacc0010e35bdca95013203e691d402900"
Jun 15 05:50:27.272667 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.272646    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jun 15 05:50:27.272984 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.272967    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerName="prom-label-proxy"
Jun 15 05:50:27.273062 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.272987    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerName="prom-label-proxy"
Jun 15 05:50:27.273062 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.273001    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77cb750c-2c2f-41b9-ada6-7021f53ea271" containerName="console"
Jun 15 05:50:27.273062 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.273010    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="77cb750c-2c2f-41b9-ada6-7021f53ea271" containerName="console"
Jun 15 05:50:27.273062 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.273025    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerName="kube-rbac-proxy-web"
Jun 15 05:50:27.273062 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.273034    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerName="kube-rbac-proxy-web"
Jun 15 05:50:27.273062 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.273047    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerName="kube-rbac-proxy-metric"
Jun 15 05:50:27.273062 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.273056    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerName="kube-rbac-proxy-metric"
Jun 15 05:50:27.273398 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.273067    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerName="config-reloader"
Jun 15 05:50:27.273398 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.273076    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerName="config-reloader"
Jun 15 05:50:27.273398 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.273087    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerName="init-config-reloader"
Jun 15 05:50:27.273398 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.273096    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerName="init-config-reloader"
Jun 15 05:50:27.273398 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.273116    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerName="kube-rbac-proxy"
Jun 15 05:50:27.273398 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.273124    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerName="kube-rbac-proxy"
Jun 15 05:50:27.273398 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.273147    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerName="alertmanager"
Jun 15 05:50:27.273398 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.273155    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerName="alertmanager"
Jun 15 05:50:27.273398 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.273250    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerName="kube-rbac-proxy-metric"
Jun 15 05:50:27.273398 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.273263    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerName="prom-label-proxy"
Jun 15 05:50:27.273398 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.273272    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="77cb750c-2c2f-41b9-ada6-7021f53ea271" containerName="console"
Jun 15 05:50:27.273398 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.273282    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerName="kube-rbac-proxy-web"
Jun 15 05:50:27.273398 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.273290    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerName="kube-rbac-proxy"
Jun 15 05:50:27.273398 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.273300    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerName="alertmanager"
Jun 15 05:50:27.273398 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.273308    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="08a5d13e-ee76-4cd0-b100-e51a330f1765" containerName="config-reloader"
Jun 15 05:50:27.277194 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.277176    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.279740 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.279720    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jun 15 05:50:27.279993 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.279971    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jun 15 05:50:27.279993 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.279989    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jun 15 05:50:27.280136 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.279972    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jun 15 05:50:27.280136 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.280068    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jun 15 05:50:27.280447 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.280431    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jun 15 05:50:27.280814 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.280798    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jun 15 05:50:27.281314 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.281103    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jun 15 05:50:27.283071 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.281827    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-n66l2\""
Jun 15 05:50:27.287831 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.287811    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jun 15 05:50:27.294874 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.294855    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jun 15 05:50:27.323977 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.323951    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08a5d13e-ee76-4cd0-b100-e51a330f1765" path="/var/lib/kubelet/pods/08a5d13e-ee76-4cd0-b100-e51a330f1765/volumes"
Jun 15 05:50:27.438575 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.438543    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/60712149-ddd7-40bc-8c65-bbccb02d229f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.438726 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.438583    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60712149-ddd7-40bc-8c65-bbccb02d229f-config-out\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.438726 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.438601    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/60712149-ddd7-40bc-8c65-bbccb02d229f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.438726 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.438644    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60712149-ddd7-40bc-8c65-bbccb02d229f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.438726 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.438671    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60712149-ddd7-40bc-8c65-bbccb02d229f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.438726 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.438692    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/60712149-ddd7-40bc-8c65-bbccb02d229f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.438900 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.438867    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60712149-ddd7-40bc-8c65-bbccb02d229f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.438933 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.438907    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60712149-ddd7-40bc-8c65-bbccb02d229f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.438966 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.438931    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60712149-ddd7-40bc-8c65-bbccb02d229f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.438998 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.438987    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60712149-ddd7-40bc-8c65-bbccb02d229f-web-config\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.439030 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.439007    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn27z\" (UniqueName: \"kubernetes.io/projected/60712149-ddd7-40bc-8c65-bbccb02d229f-kube-api-access-xn27z\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.439064 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.439037    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/60712149-ddd7-40bc-8c65-bbccb02d229f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.439064 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.439054    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/60712149-ddd7-40bc-8c65-bbccb02d229f-config-volume\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.539864 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.539821    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60712149-ddd7-40bc-8c65-bbccb02d229f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.539864 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.539862    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60712149-ddd7-40bc-8c65-bbccb02d229f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.540106 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.539880    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60712149-ddd7-40bc-8c65-bbccb02d229f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.540106 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.539907    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60712149-ddd7-40bc-8c65-bbccb02d229f-web-config\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.540106 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.539929    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn27z\" (UniqueName: \"kubernetes.io/projected/60712149-ddd7-40bc-8c65-bbccb02d229f-kube-api-access-xn27z\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.540106 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.539958    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/60712149-ddd7-40bc-8c65-bbccb02d229f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.540106 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.539985    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/60712149-ddd7-40bc-8c65-bbccb02d229f-config-volume\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.540106 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.540028    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/60712149-ddd7-40bc-8c65-bbccb02d229f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.540106 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.540054    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60712149-ddd7-40bc-8c65-bbccb02d229f-config-out\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.540106 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.540071    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/60712149-ddd7-40bc-8c65-bbccb02d229f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.540106 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.540089    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60712149-ddd7-40bc-8c65-bbccb02d229f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.540106 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.540111    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60712149-ddd7-40bc-8c65-bbccb02d229f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.540778 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.540138    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/60712149-ddd7-40bc-8c65-bbccb02d229f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.540778 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.540566    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/60712149-ddd7-40bc-8c65-bbccb02d229f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.540778 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.540752    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60712149-ddd7-40bc-8c65-bbccb02d229f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.540938 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.540812    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60712149-ddd7-40bc-8c65-bbccb02d229f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.542914 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.542890    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60712149-ddd7-40bc-8c65-bbccb02d229f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.543044 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.543029    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/60712149-ddd7-40bc-8c65-bbccb02d229f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.543419 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.543397    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60712149-ddd7-40bc-8c65-bbccb02d229f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.543544 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.543526    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60712149-ddd7-40bc-8c65-bbccb02d229f-config-out\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.543584 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.543538    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/60712149-ddd7-40bc-8c65-bbccb02d229f-config-volume\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.543584 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.543544    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/60712149-ddd7-40bc-8c65-bbccb02d229f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.543689 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.543669    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60712149-ddd7-40bc-8c65-bbccb02d229f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.543919 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.543901    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60712149-ddd7-40bc-8c65-bbccb02d229f-web-config\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.544806 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.544790    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/60712149-ddd7-40bc-8c65-bbccb02d229f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.550234 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.550210    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn27z\" (UniqueName: \"kubernetes.io/projected/60712149-ddd7-40bc-8c65-bbccb02d229f-kube-api-access-xn27z\") pod \"alertmanager-main-0\" (UID: \"60712149-ddd7-40bc-8c65-bbccb02d229f\") " pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.587220 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.587195    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jun 15 05:50:27.739465 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:27.739390    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jun 15 05:50:27.742880 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:50:27.742849    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60712149_ddd7_40bc_8c65_bbccb02d229f.slice/crio-18d9da321a499593670c8a6ce65f8353b9143b4fd5793a146a87e35001eb05b5 WatchSource:0}: Error finding container 18d9da321a499593670c8a6ce65f8353b9143b4fd5793a146a87e35001eb05b5: Status 404 returned error can't find the container with id 18d9da321a499593670c8a6ce65f8353b9143b4fd5793a146a87e35001eb05b5
Jun 15 05:50:28.202489 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:28.202457    2572 generic.go:356] "Generic (PLEG): container finished" podID="60712149-ddd7-40bc-8c65-bbccb02d229f" containerID="0de143cc598b7b252e5e02b9560c2d44c9dcc03185b3cd92500be423ff1c2fe3" exitCode=0
Jun 15 05:50:28.202852 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:28.202508    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60712149-ddd7-40bc-8c65-bbccb02d229f","Type":"ContainerDied","Data":"0de143cc598b7b252e5e02b9560c2d44c9dcc03185b3cd92500be423ff1c2fe3"}
Jun 15 05:50:28.202852 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:28.202560    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60712149-ddd7-40bc-8c65-bbccb02d229f","Type":"ContainerStarted","Data":"18d9da321a499593670c8a6ce65f8353b9143b4fd5793a146a87e35001eb05b5"}
Jun 15 05:50:29.035769 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.035727    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5cf6999874-hqcvz"]
Jun 15 05:50:29.040734 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.040703    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz"
Jun 15 05:50:29.043409 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.043387    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jun 15 05:50:29.043923 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.043886    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jun 15 05:50:29.044043 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.043942    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jun 15 05:50:29.044043 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.044011    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jun 15 05:50:29.044159 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.044019    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-6nmj4\""
Jun 15 05:50:29.044398 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.044386    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jun 15 05:50:29.051190 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.051171    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jun 15 05:50:29.052089 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.052068    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5cf6999874-hqcvz"]
Jun 15 05:50:29.155257 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.155210    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4d27f524-03f3-45ca-8026-9880e1cfc205-telemeter-client-tls\") pod \"telemeter-client-5cf6999874-hqcvz\" (UID: \"4d27f524-03f3-45ca-8026-9880e1cfc205\") " pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz"
Jun 15 05:50:29.155463 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.155273    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d27f524-03f3-45ca-8026-9880e1cfc205-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5cf6999874-hqcvz\" (UID: \"4d27f524-03f3-45ca-8026-9880e1cfc205\") " pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz"
Jun 15 05:50:29.155463 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.155306    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s82bh\" (UniqueName: \"kubernetes.io/projected/4d27f524-03f3-45ca-8026-9880e1cfc205-kube-api-access-s82bh\") pod \"telemeter-client-5cf6999874-hqcvz\" (UID: \"4d27f524-03f3-45ca-8026-9880e1cfc205\") " pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz"
Jun 15 05:50:29.155463 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.155335    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d27f524-03f3-45ca-8026-9880e1cfc205-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5cf6999874-hqcvz\" (UID: \"4d27f524-03f3-45ca-8026-9880e1cfc205\") " pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz"
Jun 15 05:50:29.155463 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.155392    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4d27f524-03f3-45ca-8026-9880e1cfc205-secret-telemeter-client\") pod \"telemeter-client-5cf6999874-hqcvz\" (UID: \"4d27f524-03f3-45ca-8026-9880e1cfc205\") " pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz"
Jun 15 05:50:29.155618 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.155469    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4d27f524-03f3-45ca-8026-9880e1cfc205-federate-client-tls\") pod \"telemeter-client-5cf6999874-hqcvz\" (UID: \"4d27f524-03f3-45ca-8026-9880e1cfc205\") " pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz"
Jun 15 05:50:29.155618 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.155538    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d27f524-03f3-45ca-8026-9880e1cfc205-metrics-client-ca\") pod \"telemeter-client-5cf6999874-hqcvz\" (UID: \"4d27f524-03f3-45ca-8026-9880e1cfc205\") " pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz"
Jun 15 05:50:29.155618 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.155573    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d27f524-03f3-45ca-8026-9880e1cfc205-serving-certs-ca-bundle\") pod \"telemeter-client-5cf6999874-hqcvz\" (UID: \"4d27f524-03f3-45ca-8026-9880e1cfc205\") " pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz"
Jun 15 05:50:29.211393 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.211337    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60712149-ddd7-40bc-8c65-bbccb02d229f","Type":"ContainerStarted","Data":"62172bb8223d1a2ac67e5ba30d0dd4cff44023bb778c6de19d182ec574832bae"}
Jun 15 05:50:29.211859 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.211445    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60712149-ddd7-40bc-8c65-bbccb02d229f","Type":"ContainerStarted","Data":"aeba678f938ddf4f3a600c80c544ca6245bdcafe94873245d05c20469acb5015"}
Jun 15 05:50:29.211859 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.211473    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60712149-ddd7-40bc-8c65-bbccb02d229f","Type":"ContainerStarted","Data":"6426f9aabfd34f8c7cbbbc2666fb15767b1b6ec5521bc194716e6a7337aed95f"}
Jun 15 05:50:29.211859 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.211500    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60712149-ddd7-40bc-8c65-bbccb02d229f","Type":"ContainerStarted","Data":"31a7802638a347752a0391b18d585596dd830a36821591afea7b0527c03e9187"}
Jun 15 05:50:29.211859 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.211526    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60712149-ddd7-40bc-8c65-bbccb02d229f","Type":"ContainerStarted","Data":"25451ff8d979277fa123fabfdddf594528af29f1b6d0656be8c5cc00c2dd7303"}
Jun 15 05:50:29.211859 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.211550    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"60712149-ddd7-40bc-8c65-bbccb02d229f","Type":"ContainerStarted","Data":"556c7e777dfc736e9c3be4ec7ba4b0086366f3b2d179f96723144385e9b3389a"}
Jun 15 05:50:29.244706 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.244639    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.225701613 podStartE2EDuration="2.244615315s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.913702ms" isStatefulPod=false podCreationTimestamp="2026-06-15 05:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:50:29.242643863 +0000 UTC m=+144.390823863" watchObservedRunningTime="2026-06-15 05:50:29.244615315 +0000 UTC m=+144.392795314"
Jun 15 05:50:29.256395 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.256340    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d27f524-03f3-45ca-8026-9880e1cfc205-metrics-client-ca\") pod \"telemeter-client-5cf6999874-hqcvz\" (UID: \"4d27f524-03f3-45ca-8026-9880e1cfc205\") " pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz"
Jun 15 05:50:29.256569 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.256419    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d27f524-03f3-45ca-8026-9880e1cfc205-serving-certs-ca-bundle\") pod \"telemeter-client-5cf6999874-hqcvz\" (UID: \"4d27f524-03f3-45ca-8026-9880e1cfc205\") " pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz"
Jun 15 05:50:29.256720 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.256687    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4d27f524-03f3-45ca-8026-9880e1cfc205-telemeter-client-tls\") pod \"telemeter-client-5cf6999874-hqcvz\" (UID: \"4d27f524-03f3-45ca-8026-9880e1cfc205\") " pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz"
Jun 15 05:50:29.256799 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.256732    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d27f524-03f3-45ca-8026-9880e1cfc205-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5cf6999874-hqcvz\" (UID: \"4d27f524-03f3-45ca-8026-9880e1cfc205\") " pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz"
Jun 15 05:50:29.256799 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.256763    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s82bh\" (UniqueName: \"kubernetes.io/projected/4d27f524-03f3-45ca-8026-9880e1cfc205-kube-api-access-s82bh\") pod \"telemeter-client-5cf6999874-hqcvz\" (UID: \"4d27f524-03f3-45ca-8026-9880e1cfc205\") " pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz"
Jun 15 05:50:29.256799 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.256790    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d27f524-03f3-45ca-8026-9880e1cfc205-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5cf6999874-hqcvz\" (UID: \"4d27f524-03f3-45ca-8026-9880e1cfc205\") " pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz"
Jun 15 05:50:29.256938 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.256818    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4d27f524-03f3-45ca-8026-9880e1cfc205-secret-telemeter-client\") pod \"telemeter-client-5cf6999874-hqcvz\" (UID: \"4d27f524-03f3-45ca-8026-9880e1cfc205\") " pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz"
Jun 15 05:50:29.256938 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.256852    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4d27f524-03f3-45ca-8026-9880e1cfc205-federate-client-tls\") pod \"telemeter-client-5cf6999874-hqcvz\" (UID: \"4d27f524-03f3-45ca-8026-9880e1cfc205\") " pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz"
Jun 15 05:50:29.257128 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.257102    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d27f524-03f3-45ca-8026-9880e1cfc205-metrics-client-ca\") pod \"telemeter-client-5cf6999874-hqcvz\" (UID: \"4d27f524-03f3-45ca-8026-9880e1cfc205\") " pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz"
Jun 15 05:50:29.257199 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.257132    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d27f524-03f3-45ca-8026-9880e1cfc205-serving-certs-ca-bundle\") pod \"telemeter-client-5cf6999874-hqcvz\" (UID: \"4d27f524-03f3-45ca-8026-9880e1cfc205\") " pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz"
Jun 15 05:50:29.257872 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.257848    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d27f524-03f3-45ca-8026-9880e1cfc205-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5cf6999874-hqcvz\" (UID: \"4d27f524-03f3-45ca-8026-9880e1cfc205\") " pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz"
Jun 15 05:50:29.259809 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.259768    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4d27f524-03f3-45ca-8026-9880e1cfc205-federate-client-tls\") pod \"telemeter-client-5cf6999874-hqcvz\" (UID: \"4d27f524-03f3-45ca-8026-9880e1cfc205\") " pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz"
Jun 15 05:50:29.260134 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.260111    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4d27f524-03f3-45ca-8026-9880e1cfc205-telemeter-client-tls\") pod \"telemeter-client-5cf6999874-hqcvz\" (UID: \"4d27f524-03f3-45ca-8026-9880e1cfc205\") " pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz"
Jun 15 05:50:29.260423 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.260404    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d27f524-03f3-45ca-8026-9880e1cfc205-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5cf6999874-hqcvz\" (UID: \"4d27f524-03f3-45ca-8026-9880e1cfc205\") " pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz"
Jun 15 05:50:29.261889 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.261863    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4d27f524-03f3-45ca-8026-9880e1cfc205-secret-telemeter-client\") pod \"telemeter-client-5cf6999874-hqcvz\" (UID: \"4d27f524-03f3-45ca-8026-9880e1cfc205\") " pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz"
Jun 15 05:50:29.270353 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.270330    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s82bh\" (UniqueName: \"kubernetes.io/projected/4d27f524-03f3-45ca-8026-9880e1cfc205-kube-api-access-s82bh\") pod \"telemeter-client-5cf6999874-hqcvz\" (UID: \"4d27f524-03f3-45ca-8026-9880e1cfc205\") " pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz"
Jun 15 05:50:29.351137 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.351054    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz"
Jun 15 05:50:29.422770 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.422736    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jun 15 05:50:29.423208 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.423016    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerName="prometheus" containerID="cri-o://70eaf93693bdf8662e2fcb41a1e19df19e8e8af5f64b047edc3d5c24cfb5488b" gracePeriod=600
Jun 15 05:50:29.423208 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.423054    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerName="thanos-sidecar" containerID="cri-o://50d6795699cf4515fdb42080aafcd623fa367ac9155e029e8044ade0c5769f28" gracePeriod=600
Jun 15 05:50:29.423208 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.423065    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerName="kube-rbac-proxy" containerID="cri-o://0419e6565f157615b49a2a9f47d3fa15efc95e5fd4339db5edd64b29e23e860b" gracePeriod=600
Jun 15 05:50:29.423208 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.423083    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerName="kube-rbac-proxy-thanos" containerID="cri-o://be2f002fa7d70c3868eb5eab904311daed52c62bd1030e1df483f16d34584f44" gracePeriod=600
Jun 15 05:50:29.423208 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.423165    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerName="kube-rbac-proxy-web" containerID="cri-o://32af28c467b33602f23c086efa5fcdc1324825499a80fe80bff358f6b665b258" gracePeriod=600
Jun 15 05:50:29.423208 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.423186    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerName="config-reloader" containerID="cri-o://80025a1efc19d179e7472309d0282d6a382fc1f37da5ea681213a950cddfc0b9" gracePeriod=600
Jun 15 05:50:29.511467 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:29.511396    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5cf6999874-hqcvz"]
Jun 15 05:50:29.514754 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:50:29.514726    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d27f524_03f3_45ca_8026_9880e1cfc205.slice/crio-be0e73b7ef7701e8e0cff363778c7cac3cd7328f09bddc18133ba049a1cc2d5f WatchSource:0}: Error finding container be0e73b7ef7701e8e0cff363778c7cac3cd7328f09bddc18133ba049a1cc2d5f: Status 404 returned error can't find the container with id be0e73b7ef7701e8e0cff363778c7cac3cd7328f09bddc18133ba049a1cc2d5f
Jun 15 05:50:30.219889 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.219854    2572 generic.go:356] "Generic (PLEG): container finished" podID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerID="be2f002fa7d70c3868eb5eab904311daed52c62bd1030e1df483f16d34584f44" exitCode=0
Jun 15 05:50:30.219889 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.219883    2572 generic.go:356] "Generic (PLEG): container finished" podID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerID="0419e6565f157615b49a2a9f47d3fa15efc95e5fd4339db5edd64b29e23e860b" exitCode=0
Jun 15 05:50:30.219889 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.219892    2572 generic.go:356] "Generic (PLEG): container finished" podID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerID="50d6795699cf4515fdb42080aafcd623fa367ac9155e029e8044ade0c5769f28" exitCode=0
Jun 15 05:50:30.219889 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.219898    2572 generic.go:356] "Generic (PLEG): container finished" podID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerID="80025a1efc19d179e7472309d0282d6a382fc1f37da5ea681213a950cddfc0b9" exitCode=0
Jun 15 05:50:30.220472 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.219903    2572 generic.go:356] "Generic (PLEG): container finished" podID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerID="70eaf93693bdf8662e2fcb41a1e19df19e8e8af5f64b047edc3d5c24cfb5488b" exitCode=0
Jun 15 05:50:30.220472 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.219928    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0dabd34-b321-41af-8732-7c7b5b8eac93","Type":"ContainerDied","Data":"be2f002fa7d70c3868eb5eab904311daed52c62bd1030e1df483f16d34584f44"}
Jun 15 05:50:30.220472 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.220013    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0dabd34-b321-41af-8732-7c7b5b8eac93","Type":"ContainerDied","Data":"0419e6565f157615b49a2a9f47d3fa15efc95e5fd4339db5edd64b29e23e860b"}
Jun 15 05:50:30.220472 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.220059    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0dabd34-b321-41af-8732-7c7b5b8eac93","Type":"ContainerDied","Data":"50d6795699cf4515fdb42080aafcd623fa367ac9155e029e8044ade0c5769f28"}
Jun 15 05:50:30.220472 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.220097    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0dabd34-b321-41af-8732-7c7b5b8eac93","Type":"ContainerDied","Data":"80025a1efc19d179e7472309d0282d6a382fc1f37da5ea681213a950cddfc0b9"}
Jun 15 05:50:30.220472 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.220138    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0dabd34-b321-41af-8732-7c7b5b8eac93","Type":"ContainerDied","Data":"70eaf93693bdf8662e2fcb41a1e19df19e8e8af5f64b047edc3d5c24cfb5488b"}
Jun 15 05:50:30.221068 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.221049    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz" event={"ID":"4d27f524-03f3-45ca-8026-9880e1cfc205","Type":"ContainerStarted","Data":"be0e73b7ef7701e8e0cff363778c7cac3cd7328f09bddc18133ba049a1cc2d5f"}
Jun 15 05:50:30.677440 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.677414    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:30.770751 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.770660    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-prometheus-k8s-tls\") pod \"e0dabd34-b321-41af-8732-7c7b5b8eac93\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") "
Jun 15 05:50:30.770751 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.770708    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vsfmk\" (UniqueName: \"kubernetes.io/projected/e0dabd34-b321-41af-8732-7c7b5b8eac93-kube-api-access-vsfmk\") pod \"e0dabd34-b321-41af-8732-7c7b5b8eac93\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") "
Jun 15 05:50:30.770751 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.770733    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-k8s-rulefiles-2\") pod \"e0dabd34-b321-41af-8732-7c7b5b8eac93\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") "
Jun 15 05:50:30.771008 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.770756    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0dabd34-b321-41af-8732-7c7b5b8eac93-tls-assets\") pod \"e0dabd34-b321-41af-8732-7c7b5b8eac93\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") "
Jun 15 05:50:30.771008 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.770779    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-configmap-serving-certs-ca-bundle\") pod \"e0dabd34-b321-41af-8732-7c7b5b8eac93\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") "
Jun 15 05:50:30.771008 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.770809    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-metrics-client-certs\") pod \"e0dabd34-b321-41af-8732-7c7b5b8eac93\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") "
Jun 15 05:50:30.771008 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.770832    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-kube-rbac-proxy\") pod \"e0dabd34-b321-41af-8732-7c7b5b8eac93\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") "
Jun 15 05:50:30.771008 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.770853    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-config\") pod \"e0dabd34-b321-41af-8732-7c7b5b8eac93\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") "
Jun 15 05:50:30.771008 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.770874    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-trusted-ca-bundle\") pod \"e0dabd34-b321-41af-8732-7c7b5b8eac93\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") "
Jun 15 05:50:30.771008 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.770896    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-k8s-db\") pod \"e0dabd34-b321-41af-8732-7c7b5b8eac93\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") "
Jun 15 05:50:30.771008 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.770923    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-k8s-rulefiles-0\") pod \"e0dabd34-b321-41af-8732-7c7b5b8eac93\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") "
Jun 15 05:50:30.771008 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.771005    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"e0dabd34-b321-41af-8732-7c7b5b8eac93\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") "
Jun 15 05:50:30.771438 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.771052    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0dabd34-b321-41af-8732-7c7b5b8eac93-config-out\") pod \"e0dabd34-b321-41af-8732-7c7b5b8eac93\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") "
Jun 15 05:50:30.771438 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.771090    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-configmap-kubelet-serving-ca-bundle\") pod \"e0dabd34-b321-41af-8732-7c7b5b8eac93\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") "
Jun 15 05:50:30.771438 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.771129    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-web-config\") pod \"e0dabd34-b321-41af-8732-7c7b5b8eac93\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") "
Jun 15 05:50:30.771438 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.771195    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-grpc-tls\") pod \"e0dabd34-b321-41af-8732-7c7b5b8eac93\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") "
Jun 15 05:50:30.771438 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.771240    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-k8s-rulefiles-1\") pod \"e0dabd34-b321-41af-8732-7c7b5b8eac93\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") "
Jun 15 05:50:30.771438 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.771268    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-thanos-prometheus-http-client-file\") pod \"e0dabd34-b321-41af-8732-7c7b5b8eac93\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") "
Jun 15 05:50:30.771438 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.771300    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"e0dabd34-b321-41af-8732-7c7b5b8eac93\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") "
Jun 15 05:50:30.771438 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.771326    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-configmap-metrics-client-ca\") pod \"e0dabd34-b321-41af-8732-7c7b5b8eac93\" (UID: \"e0dabd34-b321-41af-8732-7c7b5b8eac93\") "
Jun 15 05:50:30.771863 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.771506    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "e0dabd34-b321-41af-8732-7c7b5b8eac93" (UID: "e0dabd34-b321-41af-8732-7c7b5b8eac93"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:50:30.771863 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.771528    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "e0dabd34-b321-41af-8732-7c7b5b8eac93" (UID: "e0dabd34-b321-41af-8732-7c7b5b8eac93"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:50:30.771863 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.771773    2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:30.771863 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.771795    2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-trusted-ca-bundle\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:30.772060 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.771918    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "e0dabd34-b321-41af-8732-7c7b5b8eac93" (UID: "e0dabd34-b321-41af-8732-7c7b5b8eac93"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:50:30.772118 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.772081    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "e0dabd34-b321-41af-8732-7c7b5b8eac93" (UID: "e0dabd34-b321-41af-8732-7c7b5b8eac93"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:50:30.773042 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.772495    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "e0dabd34-b321-41af-8732-7c7b5b8eac93" (UID: "e0dabd34-b321-41af-8732-7c7b5b8eac93"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:50:30.774279 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.773964    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "e0dabd34-b321-41af-8732-7c7b5b8eac93" (UID: "e0dabd34-b321-41af-8732-7c7b5b8eac93"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 05:50:30.774279 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.774028    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "e0dabd34-b321-41af-8732-7c7b5b8eac93" (UID: "e0dabd34-b321-41af-8732-7c7b5b8eac93"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:50:30.774468 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.774391    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "e0dabd34-b321-41af-8732-7c7b5b8eac93" (UID: "e0dabd34-b321-41af-8732-7c7b5b8eac93"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:50:30.775326 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.775296    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "e0dabd34-b321-41af-8732-7c7b5b8eac93" (UID: "e0dabd34-b321-41af-8732-7c7b5b8eac93"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:50:30.775533 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.775494    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0dabd34-b321-41af-8732-7c7b5b8eac93-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "e0dabd34-b321-41af-8732-7c7b5b8eac93" (UID: "e0dabd34-b321-41af-8732-7c7b5b8eac93"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 05:50:30.775622 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.775603    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0dabd34-b321-41af-8732-7c7b5b8eac93-kube-api-access-vsfmk" (OuterVolumeSpecName: "kube-api-access-vsfmk") pod "e0dabd34-b321-41af-8732-7c7b5b8eac93" (UID: "e0dabd34-b321-41af-8732-7c7b5b8eac93"). InnerVolumeSpecName "kube-api-access-vsfmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 05:50:30.778046 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.778012    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "e0dabd34-b321-41af-8732-7c7b5b8eac93" (UID: "e0dabd34-b321-41af-8732-7c7b5b8eac93"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:50:30.778223 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.778203    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-config" (OuterVolumeSpecName: "config") pod "e0dabd34-b321-41af-8732-7c7b5b8eac93" (UID: "e0dabd34-b321-41af-8732-7c7b5b8eac93"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:50:30.778403 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.778348    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0dabd34-b321-41af-8732-7c7b5b8eac93-config-out" (OuterVolumeSpecName: "config-out") pod "e0dabd34-b321-41af-8732-7c7b5b8eac93" (UID: "e0dabd34-b321-41af-8732-7c7b5b8eac93"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 05:50:30.778403 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.778380    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "e0dabd34-b321-41af-8732-7c7b5b8eac93" (UID: "e0dabd34-b321-41af-8732-7c7b5b8eac93"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:50:30.778674 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.778652    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "e0dabd34-b321-41af-8732-7c7b5b8eac93" (UID: "e0dabd34-b321-41af-8732-7c7b5b8eac93"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:50:30.780146 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.780097    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "e0dabd34-b321-41af-8732-7c7b5b8eac93" (UID: "e0dabd34-b321-41af-8732-7c7b5b8eac93"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:50:30.780499 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.780476    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "e0dabd34-b321-41af-8732-7c7b5b8eac93" (UID: "e0dabd34-b321-41af-8732-7c7b5b8eac93"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:50:30.781178 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.781150    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "e0dabd34-b321-41af-8732-7c7b5b8eac93" (UID: "e0dabd34-b321-41af-8732-7c7b5b8eac93"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:50:30.792393 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.792366    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-web-config" (OuterVolumeSpecName: "web-config") pod "e0dabd34-b321-41af-8732-7c7b5b8eac93" (UID: "e0dabd34-b321-41af-8732-7c7b5b8eac93"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:50:30.873020 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.872993    2572 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:30.873020 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.873018    2572 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-web-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:30.873197 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.873031    2572 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-grpc-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:30.873197 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.873041    2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:30.873197 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.873049    2572 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-thanos-prometheus-http-client-file\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:30.873197 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.873058    2572 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:30.873197 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.873068    2572 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-configmap-metrics-client-ca\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:30.873197 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.873076    2572 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-prometheus-k8s-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:30.873197 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.873084    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsfmk\" (UniqueName: \"kubernetes.io/projected/e0dabd34-b321-41af-8732-7c7b5b8eac93-kube-api-access-vsfmk\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:30.873197 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.873093    2572 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e0dabd34-b321-41af-8732-7c7b5b8eac93-tls-assets\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:30.873197 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.873105    2572 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:30.873197 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.873115    2572 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-metrics-client-certs\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:30.873197 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.873123    2572 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-kube-rbac-proxy\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:30.873197 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.873131    2572 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:30.873197 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.873139    2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-k8s-db\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:30.873197 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.873147    2572 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e0dabd34-b321-41af-8732-7c7b5b8eac93-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:30.873197 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.873155    2572 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e0dabd34-b321-41af-8732-7c7b5b8eac93-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:30.873197 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:30.873164    2572 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e0dabd34-b321-41af-8732-7c7b5b8eac93-config-out\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:50:31.230466 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.230425    2572 generic.go:356] "Generic (PLEG): container finished" podID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerID="32af28c467b33602f23c086efa5fcdc1324825499a80fe80bff358f6b665b258" exitCode=0
Jun 15 05:50:31.230919 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.230493    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0dabd34-b321-41af-8732-7c7b5b8eac93","Type":"ContainerDied","Data":"32af28c467b33602f23c086efa5fcdc1324825499a80fe80bff358f6b665b258"}
Jun 15 05:50:31.230919 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.230519    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.230919 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.230573    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e0dabd34-b321-41af-8732-7c7b5b8eac93","Type":"ContainerDied","Data":"8630064b0605ce964b2b0b0ddd23abc0040dbd42285635d643cad419560782ac"}
Jun 15 05:50:31.230919 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.230625    2572 scope.go:117] "RemoveContainer" containerID="be2f002fa7d70c3868eb5eab904311daed52c62bd1030e1df483f16d34584f44"
Jun 15 05:50:31.271416 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.271385    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jun 15 05:50:31.277417 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.277387    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jun 15 05:50:31.312700 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.312670    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jun 15 05:50:31.313128 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.313111    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerName="thanos-sidecar"
Jun 15 05:50:31.313171 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.313132    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerName="thanos-sidecar"
Jun 15 05:50:31.313171 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.313141    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerName="kube-rbac-proxy"
Jun 15 05:50:31.313171 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.313146    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerName="kube-rbac-proxy"
Jun 15 05:50:31.313171 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.313157    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerName="prometheus"
Jun 15 05:50:31.313171 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.313162    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerName="prometheus"
Jun 15 05:50:31.313171 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.313172    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerName="init-config-reloader"
Jun 15 05:50:31.313342 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.313177    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerName="init-config-reloader"
Jun 15 05:50:31.313342 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.313185    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerName="config-reloader"
Jun 15 05:50:31.313342 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.313190    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerName="config-reloader"
Jun 15 05:50:31.313342 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.313199    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerName="kube-rbac-proxy-web"
Jun 15 05:50:31.313342 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.313204    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerName="kube-rbac-proxy-web"
Jun 15 05:50:31.313342 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.313211    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerName="kube-rbac-proxy-thanos"
Jun 15 05:50:31.313342 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.313216    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerName="kube-rbac-proxy-thanos"
Jun 15 05:50:31.313342 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.313270    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerName="prometheus"
Jun 15 05:50:31.313342 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.313277    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerName="thanos-sidecar"
Jun 15 05:50:31.313342 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.313283    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerName="kube-rbac-proxy"
Jun 15 05:50:31.313342 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.313289    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerName="config-reloader"
Jun 15 05:50:31.313342 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.313296    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerName="kube-rbac-proxy-thanos"
Jun 15 05:50:31.313342 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.313303    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0dabd34-b321-41af-8732-7c7b5b8eac93" containerName="kube-rbac-proxy-web"
Jun 15 05:50:31.315595 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.315576    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.316579 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.316553    2572 scope.go:117] "RemoveContainer" containerID="0419e6565f157615b49a2a9f47d3fa15efc95e5fd4339db5edd64b29e23e860b"
Jun 15 05:50:31.320512 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.320334    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jun 15 05:50:31.320512 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.320387    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vhmkr\""
Jun 15 05:50:31.320512 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.320456    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jun 15 05:50:31.320888 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.320435    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jun 15 05:50:31.321099 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.321083    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jun 15 05:50:31.321702 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.321578    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jun 15 05:50:31.321702 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.321577    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jun 15 05:50:31.321702 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.321636    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jun 15 05:50:31.321930 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.321733    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jun 15 05:50:31.321930 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.321863    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jun 15 05:50:31.321930 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.321894    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dhj5i5bidigv1\""
Jun 15 05:50:31.322093 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.321997    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jun 15 05:50:31.322291 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.322270    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jun 15 05:50:31.322853 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.322799    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jun 15 05:50:31.325100 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.325081    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jun 15 05:50:31.325494 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.325472    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0dabd34-b321-41af-8732-7c7b5b8eac93" path="/var/lib/kubelet/pods/e0dabd34-b321-41af-8732-7c7b5b8eac93/volumes"
Jun 15 05:50:31.329497 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.329450    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jun 15 05:50:31.340084 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.340064    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jun 15 05:50:31.343819 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.343798    2572 scope.go:117] "RemoveContainer" containerID="32af28c467b33602f23c086efa5fcdc1324825499a80fe80bff358f6b665b258"
Jun 15 05:50:31.351097 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.351078    2572 scope.go:117] "RemoveContainer" containerID="50d6795699cf4515fdb42080aafcd623fa367ac9155e029e8044ade0c5769f28"
Jun 15 05:50:31.361216 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.361194    2572 scope.go:117] "RemoveContainer" containerID="80025a1efc19d179e7472309d0282d6a382fc1f37da5ea681213a950cddfc0b9"
Jun 15 05:50:31.370435 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.370416    2572 scope.go:117] "RemoveContainer" containerID="70eaf93693bdf8662e2fcb41a1e19df19e8e8af5f64b047edc3d5c24cfb5488b"
Jun 15 05:50:31.384775 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.384520    2572 scope.go:117] "RemoveContainer" containerID="258a8c4669eb7412843b979eea3eec5eca6d71bcfb4fb2d8c1872d4b5082271b"
Jun 15 05:50:31.399861 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.399841    2572 scope.go:117] "RemoveContainer" containerID="be2f002fa7d70c3868eb5eab904311daed52c62bd1030e1df483f16d34584f44"
Jun 15 05:50:31.400223 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:50:31.400198    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be2f002fa7d70c3868eb5eab904311daed52c62bd1030e1df483f16d34584f44\": container with ID starting with be2f002fa7d70c3868eb5eab904311daed52c62bd1030e1df483f16d34584f44 not found: ID does not exist" containerID="be2f002fa7d70c3868eb5eab904311daed52c62bd1030e1df483f16d34584f44"
Jun 15 05:50:31.400307 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.400229    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be2f002fa7d70c3868eb5eab904311daed52c62bd1030e1df483f16d34584f44"} err="failed to get container status \"be2f002fa7d70c3868eb5eab904311daed52c62bd1030e1df483f16d34584f44\": rpc error: code = NotFound desc = could not find container \"be2f002fa7d70c3868eb5eab904311daed52c62bd1030e1df483f16d34584f44\": container with ID starting with be2f002fa7d70c3868eb5eab904311daed52c62bd1030e1df483f16d34584f44 not found: ID does not exist"
Jun 15 05:50:31.400307 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.400263    2572 scope.go:117] "RemoveContainer" containerID="0419e6565f157615b49a2a9f47d3fa15efc95e5fd4339db5edd64b29e23e860b"
Jun 15 05:50:31.400670 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:50:31.400653    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0419e6565f157615b49a2a9f47d3fa15efc95e5fd4339db5edd64b29e23e860b\": container with ID starting with 0419e6565f157615b49a2a9f47d3fa15efc95e5fd4339db5edd64b29e23e860b not found: ID does not exist" containerID="0419e6565f157615b49a2a9f47d3fa15efc95e5fd4339db5edd64b29e23e860b"
Jun 15 05:50:31.400723 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.400675    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0419e6565f157615b49a2a9f47d3fa15efc95e5fd4339db5edd64b29e23e860b"} err="failed to get container status \"0419e6565f157615b49a2a9f47d3fa15efc95e5fd4339db5edd64b29e23e860b\": rpc error: code = NotFound desc = could not find container \"0419e6565f157615b49a2a9f47d3fa15efc95e5fd4339db5edd64b29e23e860b\": container with ID starting with 0419e6565f157615b49a2a9f47d3fa15efc95e5fd4339db5edd64b29e23e860b not found: ID does not exist"
Jun 15 05:50:31.400723 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.400689    2572 scope.go:117] "RemoveContainer" containerID="32af28c467b33602f23c086efa5fcdc1324825499a80fe80bff358f6b665b258"
Jun 15 05:50:31.400924 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:50:31.400908    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32af28c467b33602f23c086efa5fcdc1324825499a80fe80bff358f6b665b258\": container with ID starting with 32af28c467b33602f23c086efa5fcdc1324825499a80fe80bff358f6b665b258 not found: ID does not exist" containerID="32af28c467b33602f23c086efa5fcdc1324825499a80fe80bff358f6b665b258"
Jun 15 05:50:31.400968 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.400929    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32af28c467b33602f23c086efa5fcdc1324825499a80fe80bff358f6b665b258"} err="failed to get container status \"32af28c467b33602f23c086efa5fcdc1324825499a80fe80bff358f6b665b258\": rpc error: code = NotFound desc = could not find container \"32af28c467b33602f23c086efa5fcdc1324825499a80fe80bff358f6b665b258\": container with ID starting with 32af28c467b33602f23c086efa5fcdc1324825499a80fe80bff358f6b665b258 not found: ID does not exist"
Jun 15 05:50:31.400968 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.400947    2572 scope.go:117] "RemoveContainer" containerID="50d6795699cf4515fdb42080aafcd623fa367ac9155e029e8044ade0c5769f28"
Jun 15 05:50:31.401239 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:50:31.401211    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50d6795699cf4515fdb42080aafcd623fa367ac9155e029e8044ade0c5769f28\": container with ID starting with 50d6795699cf4515fdb42080aafcd623fa367ac9155e029e8044ade0c5769f28 not found: ID does not exist" containerID="50d6795699cf4515fdb42080aafcd623fa367ac9155e029e8044ade0c5769f28"
Jun 15 05:50:31.401322 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.401245    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50d6795699cf4515fdb42080aafcd623fa367ac9155e029e8044ade0c5769f28"} err="failed to get container status \"50d6795699cf4515fdb42080aafcd623fa367ac9155e029e8044ade0c5769f28\": rpc error: code = NotFound desc = could not find container \"50d6795699cf4515fdb42080aafcd623fa367ac9155e029e8044ade0c5769f28\": container with ID starting with 50d6795699cf4515fdb42080aafcd623fa367ac9155e029e8044ade0c5769f28 not found: ID does not exist"
Jun 15 05:50:31.401322 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.401266    2572 scope.go:117] "RemoveContainer" containerID="80025a1efc19d179e7472309d0282d6a382fc1f37da5ea681213a950cddfc0b9"
Jun 15 05:50:31.401567 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:50:31.401551    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80025a1efc19d179e7472309d0282d6a382fc1f37da5ea681213a950cddfc0b9\": container with ID starting with 80025a1efc19d179e7472309d0282d6a382fc1f37da5ea681213a950cddfc0b9 not found: ID does not exist" containerID="80025a1efc19d179e7472309d0282d6a382fc1f37da5ea681213a950cddfc0b9"
Jun 15 05:50:31.401614 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.401571    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80025a1efc19d179e7472309d0282d6a382fc1f37da5ea681213a950cddfc0b9"} err="failed to get container status \"80025a1efc19d179e7472309d0282d6a382fc1f37da5ea681213a950cddfc0b9\": rpc error: code = NotFound desc = could not find container \"80025a1efc19d179e7472309d0282d6a382fc1f37da5ea681213a950cddfc0b9\": container with ID starting with 80025a1efc19d179e7472309d0282d6a382fc1f37da5ea681213a950cddfc0b9 not found: ID does not exist"
Jun 15 05:50:31.401614 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.401586    2572 scope.go:117] "RemoveContainer" containerID="70eaf93693bdf8662e2fcb41a1e19df19e8e8af5f64b047edc3d5c24cfb5488b"
Jun 15 05:50:31.401828 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:50:31.401811    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70eaf93693bdf8662e2fcb41a1e19df19e8e8af5f64b047edc3d5c24cfb5488b\": container with ID starting with 70eaf93693bdf8662e2fcb41a1e19df19e8e8af5f64b047edc3d5c24cfb5488b not found: ID does not exist" containerID="70eaf93693bdf8662e2fcb41a1e19df19e8e8af5f64b047edc3d5c24cfb5488b"
Jun 15 05:50:31.401868 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.401833    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70eaf93693bdf8662e2fcb41a1e19df19e8e8af5f64b047edc3d5c24cfb5488b"} err="failed to get container status \"70eaf93693bdf8662e2fcb41a1e19df19e8e8af5f64b047edc3d5c24cfb5488b\": rpc error: code = NotFound desc = could not find container \"70eaf93693bdf8662e2fcb41a1e19df19e8e8af5f64b047edc3d5c24cfb5488b\": container with ID starting with 70eaf93693bdf8662e2fcb41a1e19df19e8e8af5f64b047edc3d5c24cfb5488b not found: ID does not exist"
Jun 15 05:50:31.401868 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.401849    2572 scope.go:117] "RemoveContainer" containerID="258a8c4669eb7412843b979eea3eec5eca6d71bcfb4fb2d8c1872d4b5082271b"
Jun 15 05:50:31.402104 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:50:31.402086    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"258a8c4669eb7412843b979eea3eec5eca6d71bcfb4fb2d8c1872d4b5082271b\": container with ID starting with 258a8c4669eb7412843b979eea3eec5eca6d71bcfb4fb2d8c1872d4b5082271b not found: ID does not exist" containerID="258a8c4669eb7412843b979eea3eec5eca6d71bcfb4fb2d8c1872d4b5082271b"
Jun 15 05:50:31.402163 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.402111    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"258a8c4669eb7412843b979eea3eec5eca6d71bcfb4fb2d8c1872d4b5082271b"} err="failed to get container status \"258a8c4669eb7412843b979eea3eec5eca6d71bcfb4fb2d8c1872d4b5082271b\": rpc error: code = NotFound desc = could not find container \"258a8c4669eb7412843b979eea3eec5eca6d71bcfb4fb2d8c1872d4b5082271b\": container with ID starting with 258a8c4669eb7412843b979eea3eec5eca6d71bcfb4fb2d8c1872d4b5082271b not found: ID does not exist"
Jun 15 05:50:31.477702 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.477677    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.477823 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.477718    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.477823 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.477744    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.477823 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.477767    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.477823 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.477789    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.477823 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.477812    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.478026 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.477871    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-web-config\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.478026 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.477900    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.478026 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.477922    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-config\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.478026 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.477994    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-config-out\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.478196 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.478029    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.478196 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.478082    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.478196 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.478109    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.478196 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.478141    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.478196 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.478177    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.478196 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.478195    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jd8b\" (UniqueName: \"kubernetes.io/projected/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-kube-api-access-6jd8b\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.478431 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.478218    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.478431 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.478237    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.478431 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.478292    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.478431 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.478308    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.579181 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.579152    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-config\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.579324 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.579196    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-config-out\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.579324 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.579224    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.579324 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.579260    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.579324 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.579285    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.579324 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.579314    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.579668 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.579352    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.579668 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.579394    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jd8b\" (UniqueName: \"kubernetes.io/projected/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-kube-api-access-6jd8b\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.579668 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.579421    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.580091 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.580065    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.580187 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.580150    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.580250 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.580201    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.580250 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.580216    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.580346 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.580292    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.580346 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.580313    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.580346 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.580330    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.580525 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.580387    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.580525 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.580423    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.580525 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.580447    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.580683 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.580536    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.580683 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.580568    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.580683 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.580594    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.580683 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.580626    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-web-config\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.580683 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.580656    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.580919 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.580885    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.581132 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.581106    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.582562 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.581543    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.582562 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.582267    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-config-out\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.582562 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.582453    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.583337 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.583291    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.583908 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.583884    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.584059 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.584038    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.584600 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.584568    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-config\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.584753 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.584729    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.585133 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.585112    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.585226 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.585202    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-web-config\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.585395 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.585351    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.585489 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.585475    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.585809 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.585790    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.594270 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.594250    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jd8b\" (UniqueName: \"kubernetes.io/projected/3fb0e7d5-7654-4de4-8a00-60f36fb6c64f-kube-api-access-6jd8b\") pod \"prometheus-k8s-0\" (UID: \"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f\") " pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.630227 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.630193    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:31.782610 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:31.782571    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jun 15 05:50:31.785374 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:50:31.785330    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb0e7d5_7654_4de4_8a00_60f36fb6c64f.slice/crio-29c2ff4fc74922faa6cb44ca4c869bbe00ded7f1ab25d33aae9d7b2f1b5fbf38 WatchSource:0}: Error finding container 29c2ff4fc74922faa6cb44ca4c869bbe00ded7f1ab25d33aae9d7b2f1b5fbf38: Status 404 returned error can't find the container with id 29c2ff4fc74922faa6cb44ca4c869bbe00ded7f1ab25d33aae9d7b2f1b5fbf38
Jun 15 05:50:32.236663 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:32.236625    2572 generic.go:356] "Generic (PLEG): container finished" podID="3fb0e7d5-7654-4de4-8a00-60f36fb6c64f" containerID="3a63e75cbc861f7574e293db198cdafd71cd6de43217dd72f3a9ef94f2d538c1" exitCode=0
Jun 15 05:50:32.237065 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:32.236704    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f","Type":"ContainerDied","Data":"3a63e75cbc861f7574e293db198cdafd71cd6de43217dd72f3a9ef94f2d538c1"}
Jun 15 05:50:32.237065 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:32.236788    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f","Type":"ContainerStarted","Data":"29c2ff4fc74922faa6cb44ca4c869bbe00ded7f1ab25d33aae9d7b2f1b5fbf38"}
Jun 15 05:50:32.238961 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:32.238939    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz" event={"ID":"4d27f524-03f3-45ca-8026-9880e1cfc205","Type":"ContainerStarted","Data":"c488b74f97e7d07b3caa204d4b54ec0c0bc5e5a43bf7db5ebe4468095b8e580a"}
Jun 15 05:50:32.239063 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:32.238990    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz" event={"ID":"4d27f524-03f3-45ca-8026-9880e1cfc205","Type":"ContainerStarted","Data":"d7a8fb87ab9b1655f71c168ba1e648e85ce3b190a017e69f6bd45e99a0a57843"}
Jun 15 05:50:32.239063 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:32.239018    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz" event={"ID":"4d27f524-03f3-45ca-8026-9880e1cfc205","Type":"ContainerStarted","Data":"02cd2f3ab9e76062528c282fb3d8ee725daa18f259b56c8d9a7c7907de467c97"}
Jun 15 05:50:32.331913 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:32.331868    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5cf6999874-hqcvz" podStartSLOduration=1.508808111 podStartE2EDuration="3.331845491s" totalImagesPullingTime="1.82303738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:50:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:50:32.321390092 +0000 UTC m=+147.469570089" watchObservedRunningTime="2026-06-15 05:50:32.331845491 +0000 UTC m=+147.480025490"
Jun 15 05:50:33.252437 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:33.252328    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f","Type":"ContainerStarted","Data":"92ea3c879f8bd4d4ca9bc4587ab5b7a9368b94baf72aa0faa9c19102f38acb21"}
Jun 15 05:50:33.252836 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:33.252462    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f","Type":"ContainerStarted","Data":"c02fbdc3e3806b49648bbcf22c06b1c1b982799b609369fb8d442ea99c2faa5c"}
Jun 15 05:50:33.252836 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:33.252504    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f","Type":"ContainerStarted","Data":"b2fd9c6abaddd14e6d7a674f5929b469a8f5457e542d3f3b68b7ce66935c454b"}
Jun 15 05:50:33.252836 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:33.252541    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f","Type":"ContainerStarted","Data":"cf2c59938683b2700f52c9cc762a197e497697bc7e5237d81e1b32d495ac0faa"}
Jun 15 05:50:33.252836 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:33.252568    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f","Type":"ContainerStarted","Data":"f7ddc3e9ac9f46f22a034f0cc564d7fb27a6b95b6866ac8ab2fdf89b4e682f8e"}
Jun 15 05:50:33.252836 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:33.252604    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3fb0e7d5-7654-4de4-8a00-60f36fb6c64f","Type":"ContainerStarted","Data":"dd7e484e2dd5a9d2e324baa9a73bc732fc06ebb822d8ce7595f2d126e6acfeb1"}
Jun 15 05:50:33.292419 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:33.292345    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.27215642 podStartE2EDuration="2.292301654s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.145234ms" isStatefulPod=false podCreationTimestamp="2026-06-15 05:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:50:33.291347863 +0000 UTC m=+148.439527855" watchObservedRunningTime="2026-06-15 05:50:33.292301654 +0000 UTC m=+148.440501212"
Jun 15 05:50:36.630968 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:36.630916    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:50:44.155643 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:50:44.155608    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86b56dffcb-754hc"]
Jun 15 05:51:09.314514 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:09.314452    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-86b56dffcb-754hc" podUID="0d690ffa-0e72-4a14-8b31-618a7bc880ad" containerName="console" containerID="cri-o://f7f05636a9c9faa3d5aad5bf04ce373d1f9f3ec16c3980d240644775ac9a3694" gracePeriod=15
Jun 15 05:51:09.559038 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:09.559013    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86b56dffcb-754hc"
Jun 15 05:51:09.728937 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:09.728902    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d690ffa-0e72-4a14-8b31-618a7bc880ad-trusted-ca-bundle\") pod \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\" (UID: \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\") "
Jun 15 05:51:09.729123 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:09.728953    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtr9q\" (UniqueName: \"kubernetes.io/projected/0d690ffa-0e72-4a14-8b31-618a7bc880ad-kube-api-access-dtr9q\") pod \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\" (UID: \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\") "
Jun 15 05:51:09.729123 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:09.729008    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d690ffa-0e72-4a14-8b31-618a7bc880ad-console-config\") pod \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\" (UID: \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\") "
Jun 15 05:51:09.729123 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:09.729046    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d690ffa-0e72-4a14-8b31-618a7bc880ad-console-oauth-config\") pod \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\" (UID: \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\") "
Jun 15 05:51:09.729123 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:09.729065    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d690ffa-0e72-4a14-8b31-618a7bc880ad-service-ca\") pod \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\" (UID: \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\") "
Jun 15 05:51:09.729336 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:09.729129    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d690ffa-0e72-4a14-8b31-618a7bc880ad-console-serving-cert\") pod \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\" (UID: \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\") "
Jun 15 05:51:09.729336 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:09.729159    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d690ffa-0e72-4a14-8b31-618a7bc880ad-oauth-serving-cert\") pod \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\" (UID: \"0d690ffa-0e72-4a14-8b31-618a7bc880ad\") "
Jun 15 05:51:09.729479 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:09.729432    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d690ffa-0e72-4a14-8b31-618a7bc880ad-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0d690ffa-0e72-4a14-8b31-618a7bc880ad" (UID: "0d690ffa-0e72-4a14-8b31-618a7bc880ad"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:51:09.729543 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:09.729525    2572 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d690ffa-0e72-4a14-8b31-618a7bc880ad-trusted-ca-bundle\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:51:09.729543 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:09.729529    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d690ffa-0e72-4a14-8b31-618a7bc880ad-service-ca" (OuterVolumeSpecName: "service-ca") pod "0d690ffa-0e72-4a14-8b31-618a7bc880ad" (UID: "0d690ffa-0e72-4a14-8b31-618a7bc880ad"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:51:09.729642 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:09.729545    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d690ffa-0e72-4a14-8b31-618a7bc880ad-console-config" (OuterVolumeSpecName: "console-config") pod "0d690ffa-0e72-4a14-8b31-618a7bc880ad" (UID: "0d690ffa-0e72-4a14-8b31-618a7bc880ad"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:51:09.729746 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:09.729727    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d690ffa-0e72-4a14-8b31-618a7bc880ad-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0d690ffa-0e72-4a14-8b31-618a7bc880ad" (UID: "0d690ffa-0e72-4a14-8b31-618a7bc880ad"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 05:51:09.731476 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:09.731452    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d690ffa-0e72-4a14-8b31-618a7bc880ad-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0d690ffa-0e72-4a14-8b31-618a7bc880ad" (UID: "0d690ffa-0e72-4a14-8b31-618a7bc880ad"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:51:09.731585 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:09.731498    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d690ffa-0e72-4a14-8b31-618a7bc880ad-kube-api-access-dtr9q" (OuterVolumeSpecName: "kube-api-access-dtr9q") pod "0d690ffa-0e72-4a14-8b31-618a7bc880ad" (UID: "0d690ffa-0e72-4a14-8b31-618a7bc880ad"). InnerVolumeSpecName "kube-api-access-dtr9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 05:51:09.731585 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:09.731530    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d690ffa-0e72-4a14-8b31-618a7bc880ad-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0d690ffa-0e72-4a14-8b31-618a7bc880ad" (UID: "0d690ffa-0e72-4a14-8b31-618a7bc880ad"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 05:51:09.830967 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:09.830934    2572 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0d690ffa-0e72-4a14-8b31-618a7bc880ad-console-oauth-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:51:09.830967 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:09.830963    2572 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0d690ffa-0e72-4a14-8b31-618a7bc880ad-service-ca\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:51:09.830967 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:09.830972    2572 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0d690ffa-0e72-4a14-8b31-618a7bc880ad-console-serving-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:51:09.831195 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:09.830981    2572 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0d690ffa-0e72-4a14-8b31-618a7bc880ad-oauth-serving-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:51:09.831195 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:09.830991    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtr9q\" (UniqueName: \"kubernetes.io/projected/0d690ffa-0e72-4a14-8b31-618a7bc880ad-kube-api-access-dtr9q\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:51:09.831195 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:09.831000    2572 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0d690ffa-0e72-4a14-8b31-618a7bc880ad-console-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:51:10.397106 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:10.397067    2572 generic.go:356] "Generic (PLEG): container finished" podID="0d690ffa-0e72-4a14-8b31-618a7bc880ad" containerID="f7f05636a9c9faa3d5aad5bf04ce373d1f9f3ec16c3980d240644775ac9a3694" exitCode=0
Jun 15 05:51:10.397499 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:10.397128    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86b56dffcb-754hc"
Jun 15 05:51:10.397499 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:10.397150    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86b56dffcb-754hc" event={"ID":"0d690ffa-0e72-4a14-8b31-618a7bc880ad","Type":"ContainerDied","Data":"f7f05636a9c9faa3d5aad5bf04ce373d1f9f3ec16c3980d240644775ac9a3694"}
Jun 15 05:51:10.397499 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:10.397210    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86b56dffcb-754hc" event={"ID":"0d690ffa-0e72-4a14-8b31-618a7bc880ad","Type":"ContainerDied","Data":"185ef829d8664ad76e63abb2dafa8d3d4e4c880aee77623d8bbb39fd25472c3f"}
Jun 15 05:51:10.397499 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:10.397231    2572 scope.go:117] "RemoveContainer" containerID="f7f05636a9c9faa3d5aad5bf04ce373d1f9f3ec16c3980d240644775ac9a3694"
Jun 15 05:51:10.405673 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:10.405655    2572 scope.go:117] "RemoveContainer" containerID="f7f05636a9c9faa3d5aad5bf04ce373d1f9f3ec16c3980d240644775ac9a3694"
Jun 15 05:51:10.405920 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:51:10.405900    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7f05636a9c9faa3d5aad5bf04ce373d1f9f3ec16c3980d240644775ac9a3694\": container with ID starting with f7f05636a9c9faa3d5aad5bf04ce373d1f9f3ec16c3980d240644775ac9a3694 not found: ID does not exist" containerID="f7f05636a9c9faa3d5aad5bf04ce373d1f9f3ec16c3980d240644775ac9a3694"
Jun 15 05:51:10.405991 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:10.405927    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7f05636a9c9faa3d5aad5bf04ce373d1f9f3ec16c3980d240644775ac9a3694"} err="failed to get container status \"f7f05636a9c9faa3d5aad5bf04ce373d1f9f3ec16c3980d240644775ac9a3694\": rpc error: code = NotFound desc = could not find container \"f7f05636a9c9faa3d5aad5bf04ce373d1f9f3ec16c3980d240644775ac9a3694\": container with ID starting with f7f05636a9c9faa3d5aad5bf04ce373d1f9f3ec16c3980d240644775ac9a3694 not found: ID does not exist"
Jun 15 05:51:10.423491 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:10.423460    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-86b56dffcb-754hc"]
Jun 15 05:51:10.425381 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:10.425345    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-86b56dffcb-754hc"]
Jun 15 05:51:11.323049 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:11.323017    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d690ffa-0e72-4a14-8b31-618a7bc880ad" path="/var/lib/kubelet/pods/0d690ffa-0e72-4a14-8b31-618a7bc880ad/volumes"
Jun 15 05:51:31.630932 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:31.630893    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:51:31.646310 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:31.646285    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:51:32.497284 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:51:32.497251    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jun 15 05:53:05.268826 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:53:05.268796    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 05:53:05.269306 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:53:05.268802    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 05:53:05.275800 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:53:05.275782    2572 kubelet.go:1648] "Image garbage collection succeeded"
Jun 15 05:55:35.900082 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:35.899998    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-866d768b47-vq4j7"]
Jun 15 05:55:35.900531 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:35.900333    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d690ffa-0e72-4a14-8b31-618a7bc880ad" containerName="console"
Jun 15 05:55:35.900531 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:35.900344    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d690ffa-0e72-4a14-8b31-618a7bc880ad" containerName="console"
Jun 15 05:55:35.900531 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:35.900408    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d690ffa-0e72-4a14-8b31-618a7bc880ad" containerName="console"
Jun 15 05:55:35.903404 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:35.903389    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-866d768b47-vq4j7"
Jun 15 05:55:35.906142 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:35.906114    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jun 15 05:55:35.909240 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:35.909038    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jun 15 05:55:35.909240 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:35.909081    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jun 15 05:55:35.909657 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:35.909635    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qvwbv\""
Jun 15 05:55:35.909980 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:35.909958    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jun 15 05:55:35.910636 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:35.910611    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jun 15 05:55:35.910754 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:35.910736    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jun 15 05:55:35.910823 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:35.910699    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jun 15 05:55:35.910823 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:35.910660    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jun 15 05:55:35.919509 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:35.919489    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-866d768b47-vq4j7"]
Jun 15 05:55:36.020877 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:36.020844    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27b5148e-64d5-4397-bb36-d031b4d146e8-console-config\") pod \"console-866d768b47-vq4j7\" (UID: \"27b5148e-64d5-4397-bb36-d031b4d146e8\") " pod="openshift-console/console-866d768b47-vq4j7"
Jun 15 05:55:36.021034 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:36.020889    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27b5148e-64d5-4397-bb36-d031b4d146e8-console-oauth-config\") pod \"console-866d768b47-vq4j7\" (UID: \"27b5148e-64d5-4397-bb36-d031b4d146e8\") " pod="openshift-console/console-866d768b47-vq4j7"
Jun 15 05:55:36.021034 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:36.020918    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27b5148e-64d5-4397-bb36-d031b4d146e8-console-serving-cert\") pod \"console-866d768b47-vq4j7\" (UID: \"27b5148e-64d5-4397-bb36-d031b4d146e8\") " pod="openshift-console/console-866d768b47-vq4j7"
Jun 15 05:55:36.021034 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:36.020979    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b5148e-64d5-4397-bb36-d031b4d146e8-trusted-ca-bundle\") pod \"console-866d768b47-vq4j7\" (UID: \"27b5148e-64d5-4397-bb36-d031b4d146e8\") " pod="openshift-console/console-866d768b47-vq4j7"
Jun 15 05:55:36.021034 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:36.021005    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27b5148e-64d5-4397-bb36-d031b4d146e8-service-ca\") pod \"console-866d768b47-vq4j7\" (UID: \"27b5148e-64d5-4397-bb36-d031b4d146e8\") " pod="openshift-console/console-866d768b47-vq4j7"
Jun 15 05:55:36.021170 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:36.021043    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27b5148e-64d5-4397-bb36-d031b4d146e8-oauth-serving-cert\") pod \"console-866d768b47-vq4j7\" (UID: \"27b5148e-64d5-4397-bb36-d031b4d146e8\") " pod="openshift-console/console-866d768b47-vq4j7"
Jun 15 05:55:36.021170 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:36.021091    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvddp\" (UniqueName: \"kubernetes.io/projected/27b5148e-64d5-4397-bb36-d031b4d146e8-kube-api-access-zvddp\") pod \"console-866d768b47-vq4j7\" (UID: \"27b5148e-64d5-4397-bb36-d031b4d146e8\") " pod="openshift-console/console-866d768b47-vq4j7"
Jun 15 05:55:36.121841 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:36.121794    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27b5148e-64d5-4397-bb36-d031b4d146e8-oauth-serving-cert\") pod \"console-866d768b47-vq4j7\" (UID: \"27b5148e-64d5-4397-bb36-d031b4d146e8\") " pod="openshift-console/console-866d768b47-vq4j7"
Jun 15 05:55:36.121841 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:36.121850    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvddp\" (UniqueName: \"kubernetes.io/projected/27b5148e-64d5-4397-bb36-d031b4d146e8-kube-api-access-zvddp\") pod \"console-866d768b47-vq4j7\" (UID: \"27b5148e-64d5-4397-bb36-d031b4d146e8\") " pod="openshift-console/console-866d768b47-vq4j7"
Jun 15 05:55:36.122084 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:36.121885    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27b5148e-64d5-4397-bb36-d031b4d146e8-console-config\") pod \"console-866d768b47-vq4j7\" (UID: \"27b5148e-64d5-4397-bb36-d031b4d146e8\") " pod="openshift-console/console-866d768b47-vq4j7"
Jun 15 05:55:36.122084 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:36.121920    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27b5148e-64d5-4397-bb36-d031b4d146e8-console-oauth-config\") pod \"console-866d768b47-vq4j7\" (UID: \"27b5148e-64d5-4397-bb36-d031b4d146e8\") " pod="openshift-console/console-866d768b47-vq4j7"
Jun 15 05:55:36.122084 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:36.121946    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27b5148e-64d5-4397-bb36-d031b4d146e8-console-serving-cert\") pod \"console-866d768b47-vq4j7\" (UID: \"27b5148e-64d5-4397-bb36-d031b4d146e8\") " pod="openshift-console/console-866d768b47-vq4j7"
Jun 15 05:55:36.122084 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:36.121992    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b5148e-64d5-4397-bb36-d031b4d146e8-trusted-ca-bundle\") pod \"console-866d768b47-vq4j7\" (UID: \"27b5148e-64d5-4397-bb36-d031b4d146e8\") " pod="openshift-console/console-866d768b47-vq4j7"
Jun 15 05:55:36.122084 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:36.122015    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27b5148e-64d5-4397-bb36-d031b4d146e8-service-ca\") pod \"console-866d768b47-vq4j7\" (UID: \"27b5148e-64d5-4397-bb36-d031b4d146e8\") " pod="openshift-console/console-866d768b47-vq4j7"
Jun 15 05:55:36.122657 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:36.122629    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27b5148e-64d5-4397-bb36-d031b4d146e8-service-ca\") pod \"console-866d768b47-vq4j7\" (UID: \"27b5148e-64d5-4397-bb36-d031b4d146e8\") " pod="openshift-console/console-866d768b47-vq4j7"
Jun 15 05:55:36.122657 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:36.122639    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27b5148e-64d5-4397-bb36-d031b4d146e8-oauth-serving-cert\") pod \"console-866d768b47-vq4j7\" (UID: \"27b5148e-64d5-4397-bb36-d031b4d146e8\") " pod="openshift-console/console-866d768b47-vq4j7"
Jun 15 05:55:36.122779 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:36.122684    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27b5148e-64d5-4397-bb36-d031b4d146e8-console-config\") pod \"console-866d768b47-vq4j7\" (UID: \"27b5148e-64d5-4397-bb36-d031b4d146e8\") " pod="openshift-console/console-866d768b47-vq4j7"
Jun 15 05:55:36.122835 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:36.122818    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b5148e-64d5-4397-bb36-d031b4d146e8-trusted-ca-bundle\") pod \"console-866d768b47-vq4j7\" (UID: \"27b5148e-64d5-4397-bb36-d031b4d146e8\") " pod="openshift-console/console-866d768b47-vq4j7"
Jun 15 05:55:36.124346 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:36.124325    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27b5148e-64d5-4397-bb36-d031b4d146e8-console-oauth-config\") pod \"console-866d768b47-vq4j7\" (UID: \"27b5148e-64d5-4397-bb36-d031b4d146e8\") " pod="openshift-console/console-866d768b47-vq4j7"
Jun 15 05:55:36.124462 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:36.124446    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27b5148e-64d5-4397-bb36-d031b4d146e8-console-serving-cert\") pod \"console-866d768b47-vq4j7\" (UID: \"27b5148e-64d5-4397-bb36-d031b4d146e8\") " pod="openshift-console/console-866d768b47-vq4j7"
Jun 15 05:55:36.132121 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:36.132091    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvddp\" (UniqueName: \"kubernetes.io/projected/27b5148e-64d5-4397-bb36-d031b4d146e8-kube-api-access-zvddp\") pod \"console-866d768b47-vq4j7\" (UID: \"27b5148e-64d5-4397-bb36-d031b4d146e8\") " pod="openshift-console/console-866d768b47-vq4j7"
Jun 15 05:55:36.213750 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:36.213662    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-866d768b47-vq4j7"
Jun 15 05:55:36.363703 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:36.363673    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-866d768b47-vq4j7"]
Jun 15 05:55:36.366642 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:55:36.366617    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b5148e_64d5_4397_bb36_d031b4d146e8.slice/crio-694c43d0b117ce2b8b17322dc55d180f78c746589cf06d8fb90731f8e46ad99f WatchSource:0}: Error finding container 694c43d0b117ce2b8b17322dc55d180f78c746589cf06d8fb90731f8e46ad99f: Status 404 returned error can't find the container with id 694c43d0b117ce2b8b17322dc55d180f78c746589cf06d8fb90731f8e46ad99f
Jun 15 05:55:36.427556 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:36.427527    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-866d768b47-vq4j7" event={"ID":"27b5148e-64d5-4397-bb36-d031b4d146e8","Type":"ContainerStarted","Data":"694c43d0b117ce2b8b17322dc55d180f78c746589cf06d8fb90731f8e46ad99f"}
Jun 15 05:55:37.432938 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:37.432901    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-866d768b47-vq4j7" event={"ID":"27b5148e-64d5-4397-bb36-d031b4d146e8","Type":"ContainerStarted","Data":"2bfd5607acfd76b2f4ce31d9509ea18db2e6b4f6afeba1e04f8a1fd9eb25154c"}
Jun 15 05:55:37.453423 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:37.453373    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-866d768b47-vq4j7" podStartSLOduration=2.45334006 podStartE2EDuration="2.45334006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:55:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:55:37.451857121 +0000 UTC m=+452.600037117" watchObservedRunningTime="2026-06-15 05:55:37.45334006 +0000 UTC m=+452.601520057"
Jun 15 05:55:46.213878 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:46.213829    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-866d768b47-vq4j7"
Jun 15 05:55:46.214278 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:46.213898    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-866d768b47-vq4j7"
Jun 15 05:55:46.218654 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:46.218632    2572 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-866d768b47-vq4j7"
Jun 15 05:55:46.471092 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:55:46.471010    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-866d768b47-vq4j7"
Jun 15 05:56:13.607479 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:13.607444    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-zt9mh"]
Jun 15 05:56:13.610601 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:13.610578    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-zt9mh"
Jun 15 05:56:13.612741 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:13.612721    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jun 15 05:56:13.612850 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:13.612747    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-jsdld\""
Jun 15 05:56:13.613336 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:13.613317    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jun 15 05:56:13.613444 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:13.613335    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jun 15 05:56:13.624341 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:13.624319    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-vqqhs"]
Jun 15 05:56:13.628089 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:13.628069    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-zt9mh"]
Jun 15 05:56:13.628192 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:13.628155    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-vqqhs"
Jun 15 05:56:13.630348 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:13.630325    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jun 15 05:56:13.630463 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:13.630382    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-fv2gq\""
Jun 15 05:56:13.636773 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:13.636755    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-vqqhs"]
Jun 15 05:56:13.749442 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:13.749411    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4lpr\" (UniqueName: \"kubernetes.io/projected/93fa76e3-706a-4b9a-bc8d-2ffb5686e566-kube-api-access-k4lpr\") pod \"odh-model-controller-84b54b7f97-vqqhs\" (UID: \"93fa76e3-706a-4b9a-bc8d-2ffb5686e566\") " pod="kserve/odh-model-controller-84b54b7f97-vqqhs"
Jun 15 05:56:13.749442 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:13.749447    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9kcv\" (UniqueName: \"kubernetes.io/projected/ba044465-40ed-4fe8-ab1a-fee1bc6065c9-kube-api-access-m9kcv\") pod \"model-serving-api-fd65d7d6-zt9mh\" (UID: \"ba044465-40ed-4fe8-ab1a-fee1bc6065c9\") " pod="kserve/model-serving-api-fd65d7d6-zt9mh"
Jun 15 05:56:13.749668 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:13.749517    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba044465-40ed-4fe8-ab1a-fee1bc6065c9-tls-certs\") pod \"model-serving-api-fd65d7d6-zt9mh\" (UID: \"ba044465-40ed-4fe8-ab1a-fee1bc6065c9\") " pod="kserve/model-serving-api-fd65d7d6-zt9mh"
Jun 15 05:56:13.749668 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:13.749539    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93fa76e3-706a-4b9a-bc8d-2ffb5686e566-cert\") pod \"odh-model-controller-84b54b7f97-vqqhs\" (UID: \"93fa76e3-706a-4b9a-bc8d-2ffb5686e566\") " pod="kserve/odh-model-controller-84b54b7f97-vqqhs"
Jun 15 05:56:13.850533 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:13.850497    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4lpr\" (UniqueName: \"kubernetes.io/projected/93fa76e3-706a-4b9a-bc8d-2ffb5686e566-kube-api-access-k4lpr\") pod \"odh-model-controller-84b54b7f97-vqqhs\" (UID: \"93fa76e3-706a-4b9a-bc8d-2ffb5686e566\") " pod="kserve/odh-model-controller-84b54b7f97-vqqhs"
Jun 15 05:56:13.850533 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:13.850534    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9kcv\" (UniqueName: \"kubernetes.io/projected/ba044465-40ed-4fe8-ab1a-fee1bc6065c9-kube-api-access-m9kcv\") pod \"model-serving-api-fd65d7d6-zt9mh\" (UID: \"ba044465-40ed-4fe8-ab1a-fee1bc6065c9\") " pod="kserve/model-serving-api-fd65d7d6-zt9mh"
Jun 15 05:56:13.850796 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:13.850566    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba044465-40ed-4fe8-ab1a-fee1bc6065c9-tls-certs\") pod \"model-serving-api-fd65d7d6-zt9mh\" (UID: \"ba044465-40ed-4fe8-ab1a-fee1bc6065c9\") " pod="kserve/model-serving-api-fd65d7d6-zt9mh"
Jun 15 05:56:13.850796 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:13.850581    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93fa76e3-706a-4b9a-bc8d-2ffb5686e566-cert\") pod \"odh-model-controller-84b54b7f97-vqqhs\" (UID: \"93fa76e3-706a-4b9a-bc8d-2ffb5686e566\") " pod="kserve/odh-model-controller-84b54b7f97-vqqhs"
Jun 15 05:56:13.852967 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:13.852944    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba044465-40ed-4fe8-ab1a-fee1bc6065c9-tls-certs\") pod \"model-serving-api-fd65d7d6-zt9mh\" (UID: \"ba044465-40ed-4fe8-ab1a-fee1bc6065c9\") " pod="kserve/model-serving-api-fd65d7d6-zt9mh"
Jun 15 05:56:13.853126 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:13.853104    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93fa76e3-706a-4b9a-bc8d-2ffb5686e566-cert\") pod \"odh-model-controller-84b54b7f97-vqqhs\" (UID: \"93fa76e3-706a-4b9a-bc8d-2ffb5686e566\") " pod="kserve/odh-model-controller-84b54b7f97-vqqhs"
Jun 15 05:56:13.860072 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:13.860017    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9kcv\" (UniqueName: \"kubernetes.io/projected/ba044465-40ed-4fe8-ab1a-fee1bc6065c9-kube-api-access-m9kcv\") pod \"model-serving-api-fd65d7d6-zt9mh\" (UID: \"ba044465-40ed-4fe8-ab1a-fee1bc6065c9\") " pod="kserve/model-serving-api-fd65d7d6-zt9mh"
Jun 15 05:56:13.860167 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:13.860076    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4lpr\" (UniqueName: \"kubernetes.io/projected/93fa76e3-706a-4b9a-bc8d-2ffb5686e566-kube-api-access-k4lpr\") pod \"odh-model-controller-84b54b7f97-vqqhs\" (UID: \"93fa76e3-706a-4b9a-bc8d-2ffb5686e566\") " pod="kserve/odh-model-controller-84b54b7f97-vqqhs"
Jun 15 05:56:13.922742 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:13.922713    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-zt9mh"
Jun 15 05:56:13.938961 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:13.938930    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-vqqhs"
Jun 15 05:56:14.063773 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:14.063751    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-zt9mh"]
Jun 15 05:56:14.066593 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:56:14.066557    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba044465_40ed_4fe8_ab1a_fee1bc6065c9.slice/crio-ab8eaf036484d573b96948d1ffafc0d6a35d7c318c9d73ba2f3286eb7c51077e WatchSource:0}: Error finding container ab8eaf036484d573b96948d1ffafc0d6a35d7c318c9d73ba2f3286eb7c51077e: Status 404 returned error can't find the container with id ab8eaf036484d573b96948d1ffafc0d6a35d7c318c9d73ba2f3286eb7c51077e
Jun 15 05:56:14.068336 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:14.068321    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 15 05:56:14.080692 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:14.080666    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-vqqhs"]
Jun 15 05:56:14.082862 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:56:14.082833    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93fa76e3_706a_4b9a_bc8d_2ffb5686e566.slice/crio-f0b91bdc7baea57dac71af53a65478e4f704e5c33ee54ea1becf270cafc05bfe WatchSource:0}: Error finding container f0b91bdc7baea57dac71af53a65478e4f704e5c33ee54ea1becf270cafc05bfe: Status 404 returned error can't find the container with id f0b91bdc7baea57dac71af53a65478e4f704e5c33ee54ea1becf270cafc05bfe
Jun 15 05:56:14.572886 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:14.572849    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-vqqhs" event={"ID":"93fa76e3-706a-4b9a-bc8d-2ffb5686e566","Type":"ContainerStarted","Data":"f0b91bdc7baea57dac71af53a65478e4f704e5c33ee54ea1becf270cafc05bfe"}
Jun 15 05:56:14.573945 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:14.573917    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-zt9mh" event={"ID":"ba044465-40ed-4fe8-ab1a-fee1bc6065c9","Type":"ContainerStarted","Data":"ab8eaf036484d573b96948d1ffafc0d6a35d7c318c9d73ba2f3286eb7c51077e"}
Jun 15 05:56:18.594284 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:18.594247    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-vqqhs" event={"ID":"93fa76e3-706a-4b9a-bc8d-2ffb5686e566","Type":"ContainerStarted","Data":"eb3164776515b16c88cab400b92465df14dde6607bff8e6cec8d695332d130f4"}
Jun 15 05:56:18.594699 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:18.594340    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-vqqhs"
Jun 15 05:56:18.595654 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:18.595633    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-zt9mh" event={"ID":"ba044465-40ed-4fe8-ab1a-fee1bc6065c9","Type":"ContainerStarted","Data":"79ec62d08155c9c1e3913a51853067ae7758fd569c484c5570114589a14a1156"}
Jun 15 05:56:18.595770 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:18.595751    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-zt9mh"
Jun 15 05:56:18.612200 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:18.612149    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-vqqhs" podStartSLOduration=1.7655383420000001 podStartE2EDuration="5.61212976s" totalImagesPullingTime="3.846591418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:56:18.611713936 +0000 UTC m=+493.759893932" watchObservedRunningTime="2026-06-15 05:56:18.61212976 +0000 UTC m=+493.760309760"
Jun 15 05:56:18.634736 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:18.634684    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-zt9mh" podStartSLOduration=1.776445002 podStartE2EDuration="5.634666679s" totalImagesPullingTime="3.858221677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:56:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:56:18.632961596 +0000 UTC m=+493.781141593" watchObservedRunningTime="2026-06-15 05:56:18.634666679 +0000 UTC m=+493.782846691"
Jun 15 05:56:29.602032 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:29.602000    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-vqqhs"
Jun 15 05:56:29.603992 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:29.603964    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-zt9mh"
Jun 15 05:56:30.497910 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:30.497880    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-nrvh6"]
Jun 15 05:56:30.501599 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:30.501582    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-nrvh6"
Jun 15 05:56:30.505566 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:30.505547    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-wtsnv\""
Jun 15 05:56:30.506059 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:30.506043    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jun 15 05:56:30.521524 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:30.521499    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-nrvh6"]
Jun 15 05:56:30.596887 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:30.596855    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wt66\" (UniqueName: \"kubernetes.io/projected/8ccda16a-657f-4bca-8e42-e84ce0b5c4a7-kube-api-access-6wt66\") pod \"s3-init-nrvh6\" (UID: \"8ccda16a-657f-4bca-8e42-e84ce0b5c4a7\") " pod="kserve/s3-init-nrvh6"
Jun 15 05:56:30.597061 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:30.596923    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ccda16a-657f-4bca-8e42-e84ce0b5c4a7-model-cache\") pod \"s3-init-nrvh6\" (UID: \"8ccda16a-657f-4bca-8e42-e84ce0b5c4a7\") " pod="kserve/s3-init-nrvh6"
Jun 15 05:56:30.698053 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:30.698015    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ccda16a-657f-4bca-8e42-e84ce0b5c4a7-model-cache\") pod \"s3-init-nrvh6\" (UID: \"8ccda16a-657f-4bca-8e42-e84ce0b5c4a7\") " pod="kserve/s3-init-nrvh6"
Jun 15 05:56:30.698571 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:30.698081    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wt66\" (UniqueName: \"kubernetes.io/projected/8ccda16a-657f-4bca-8e42-e84ce0b5c4a7-kube-api-access-6wt66\") pod \"s3-init-nrvh6\" (UID: \"8ccda16a-657f-4bca-8e42-e84ce0b5c4a7\") " pod="kserve/s3-init-nrvh6"
Jun 15 05:56:30.698571 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:30.698506    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ccda16a-657f-4bca-8e42-e84ce0b5c4a7-model-cache\") pod \"s3-init-nrvh6\" (UID: \"8ccda16a-657f-4bca-8e42-e84ce0b5c4a7\") " pod="kserve/s3-init-nrvh6"
Jun 15 05:56:30.709689 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:30.709661    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wt66\" (UniqueName: \"kubernetes.io/projected/8ccda16a-657f-4bca-8e42-e84ce0b5c4a7-kube-api-access-6wt66\") pod \"s3-init-nrvh6\" (UID: \"8ccda16a-657f-4bca-8e42-e84ce0b5c4a7\") " pod="kserve/s3-init-nrvh6"
Jun 15 05:56:30.826425 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:30.826324    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-nrvh6"
Jun 15 05:56:30.962378 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:30.962322    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-nrvh6"]
Jun 15 05:56:30.966668 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:56:30.966639    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ccda16a_657f_4bca_8e42_e84ce0b5c4a7.slice/crio-5e3486a602192b7bcf9fa3ce5502c06a1497c08f4cb1f75511152e8061fcbb5a WatchSource:0}: Error finding container 5e3486a602192b7bcf9fa3ce5502c06a1497c08f4cb1f75511152e8061fcbb5a: Status 404 returned error can't find the container with id 5e3486a602192b7bcf9fa3ce5502c06a1497c08f4cb1f75511152e8061fcbb5a
Jun 15 05:56:31.655790 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:31.655758    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nrvh6" event={"ID":"8ccda16a-657f-4bca-8e42-e84ce0b5c4a7","Type":"ContainerStarted","Data":"5e3486a602192b7bcf9fa3ce5502c06a1497c08f4cb1f75511152e8061fcbb5a"}
Jun 15 05:56:35.675637 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:35.675602    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nrvh6" event={"ID":"8ccda16a-657f-4bca-8e42-e84ce0b5c4a7","Type":"ContainerStarted","Data":"53e1f9c0cd958e44eb8f058790dabaf3dab5784e46ceed883bfee235a81126c6"}
Jun 15 05:56:36.680903 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:36.680866    2572 generic.go:356] "Generic (PLEG): container finished" podID="8ccda16a-657f-4bca-8e42-e84ce0b5c4a7" containerID="53e1f9c0cd958e44eb8f058790dabaf3dab5784e46ceed883bfee235a81126c6" exitCode=0
Jun 15 05:56:36.681465 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:36.680929    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nrvh6" event={"ID":"8ccda16a-657f-4bca-8e42-e84ce0b5c4a7","Type":"ContainerDied","Data":"53e1f9c0cd958e44eb8f058790dabaf3dab5784e46ceed883bfee235a81126c6"}
Jun 15 05:56:41.707886 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:41.707852    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nrvh6" event={"ID":"8ccda16a-657f-4bca-8e42-e84ce0b5c4a7","Type":"ContainerStarted","Data":"3ffa61d50a4da43270011d251dba0a356df85172c7a98658327ad19f1d04af2e"}
Jun 15 05:56:41.728637 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:56:41.728591    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-nrvh6" podStartSLOduration=1.9437879599999999 podStartE2EDuration="11.728574388s" totalImagesPullingTime="8.366240155s" totalInitContainerRuntime="1.418546273s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:56:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:56:41.727666971 +0000 UTC m=+516.875846969" watchObservedRunningTime="2026-06-15 05:56:41.728574388 +0000 UTC m=+516.876754384"
Jun 15 05:57:46.980668 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:57:46.980632    2572 generic.go:356] "Generic (PLEG): container finished" podID="8ccda16a-657f-4bca-8e42-e84ce0b5c4a7" containerID="3ffa61d50a4da43270011d251dba0a356df85172c7a98658327ad19f1d04af2e" exitCode=0
Jun 15 05:57:46.981168 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:57:46.980706    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nrvh6" event={"ID":"8ccda16a-657f-4bca-8e42-e84ce0b5c4a7","Type":"ContainerDied","Data":"3ffa61d50a4da43270011d251dba0a356df85172c7a98658327ad19f1d04af2e"}
Jun 15 05:57:48.110859 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:57:48.110835    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-nrvh6"
Jun 15 05:57:48.192471 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:57:48.192443    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ccda16a-657f-4bca-8e42-e84ce0b5c4a7-model-cache\") pod \"8ccda16a-657f-4bca-8e42-e84ce0b5c4a7\" (UID: \"8ccda16a-657f-4bca-8e42-e84ce0b5c4a7\") "
Jun 15 05:57:48.192656 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:57:48.192492    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wt66\" (UniqueName: \"kubernetes.io/projected/8ccda16a-657f-4bca-8e42-e84ce0b5c4a7-kube-api-access-6wt66\") pod \"8ccda16a-657f-4bca-8e42-e84ce0b5c4a7\" (UID: \"8ccda16a-657f-4bca-8e42-e84ce0b5c4a7\") "
Jun 15 05:57:48.194719 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:57:48.194691    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ccda16a-657f-4bca-8e42-e84ce0b5c4a7-kube-api-access-6wt66" (OuterVolumeSpecName: "kube-api-access-6wt66") pod "8ccda16a-657f-4bca-8e42-e84ce0b5c4a7" (UID: "8ccda16a-657f-4bca-8e42-e84ce0b5c4a7"). InnerVolumeSpecName "kube-api-access-6wt66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 05:57:48.214898 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:57:48.214873    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ccda16a-657f-4bca-8e42-e84ce0b5c4a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "8ccda16a-657f-4bca-8e42-e84ce0b5c4a7" (UID: "8ccda16a-657f-4bca-8e42-e84ce0b5c4a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 05:57:48.293140 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:57:48.293061    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8ccda16a-657f-4bca-8e42-e84ce0b5c4a7-model-cache\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:57:48.293140 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:57:48.293091    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wt66\" (UniqueName: \"kubernetes.io/projected/8ccda16a-657f-4bca-8e42-e84ce0b5c4a7-kube-api-access-6wt66\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:57:48.990042 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:57:48.990013    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-nrvh6"
Jun 15 05:57:48.990224 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:57:48.990011    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-nrvh6" event={"ID":"8ccda16a-657f-4bca-8e42-e84ce0b5c4a7","Type":"ContainerDied","Data":"5e3486a602192b7bcf9fa3ce5502c06a1497c08f4cb1f75511152e8061fcbb5a"}
Jun 15 05:57:48.990224 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:57:48.990122    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e3486a602192b7bcf9fa3ce5502c06a1497c08f4cb1f75511152e8061fcbb5a"
Jun 15 05:58:05.300545 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:05.300517    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 05:58:05.302175 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:05.302142    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 05:58:23.545602 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:23.545526    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-2wwsz"]
Jun 15 05:58:23.546076 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:23.545912    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ccda16a-657f-4bca-8e42-e84ce0b5c4a7" containerName="s3-init"
Jun 15 05:58:23.546076 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:23.545926    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ccda16a-657f-4bca-8e42-e84ce0b5c4a7" containerName="s3-init"
Jun 15 05:58:23.546076 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:23.545940    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ccda16a-657f-4bca-8e42-e84ce0b5c4a7" containerName="download-hf-model"
Jun 15 05:58:23.546076 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:23.545946    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ccda16a-657f-4bca-8e42-e84ce0b5c4a7" containerName="download-hf-model"
Jun 15 05:58:23.546076 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:23.546001    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ccda16a-657f-4bca-8e42-e84ce0b5c4a7" containerName="s3-init"
Jun 15 05:58:23.549053 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:23.549036    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-2wwsz"
Jun 15 05:58:23.551902 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:23.551885    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jun 15 05:58:23.552382 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:23.552370    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-wtsnv\""
Jun 15 05:58:23.560346 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:23.560327    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-2wwsz"]
Jun 15 05:58:23.596923 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:23.596894    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c2c0136-4539-4589-b135-969530039e2e-model-cache\") pod \"s3-tls-init-custom-2wwsz\" (UID: \"9c2c0136-4539-4589-b135-969530039e2e\") " pod="kserve/s3-tls-init-custom-2wwsz"
Jun 15 05:58:23.596923 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:23.596924    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxv26\" (UniqueName: \"kubernetes.io/projected/9c2c0136-4539-4589-b135-969530039e2e-kube-api-access-jxv26\") pod \"s3-tls-init-custom-2wwsz\" (UID: \"9c2c0136-4539-4589-b135-969530039e2e\") " pod="kserve/s3-tls-init-custom-2wwsz"
Jun 15 05:58:23.698184 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:23.698148    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c2c0136-4539-4589-b135-969530039e2e-model-cache\") pod \"s3-tls-init-custom-2wwsz\" (UID: \"9c2c0136-4539-4589-b135-969530039e2e\") " pod="kserve/s3-tls-init-custom-2wwsz"
Jun 15 05:58:23.698184 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:23.698186    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxv26\" (UniqueName: \"kubernetes.io/projected/9c2c0136-4539-4589-b135-969530039e2e-kube-api-access-jxv26\") pod \"s3-tls-init-custom-2wwsz\" (UID: \"9c2c0136-4539-4589-b135-969530039e2e\") " pod="kserve/s3-tls-init-custom-2wwsz"
Jun 15 05:58:23.698574 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:23.698555    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c2c0136-4539-4589-b135-969530039e2e-model-cache\") pod \"s3-tls-init-custom-2wwsz\" (UID: \"9c2c0136-4539-4589-b135-969530039e2e\") " pod="kserve/s3-tls-init-custom-2wwsz"
Jun 15 05:58:23.709947 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:23.709924    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxv26\" (UniqueName: \"kubernetes.io/projected/9c2c0136-4539-4589-b135-969530039e2e-kube-api-access-jxv26\") pod \"s3-tls-init-custom-2wwsz\" (UID: \"9c2c0136-4539-4589-b135-969530039e2e\") " pod="kserve/s3-tls-init-custom-2wwsz"
Jun 15 05:58:23.867908 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:23.867819    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-2wwsz"
Jun 15 05:58:24.001228 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:24.001204    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-2wwsz"]
Jun 15 05:58:24.003314 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:58:24.003288    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c2c0136_4539_4589_b135_969530039e2e.slice/crio-9db27b7d34ef076f21d703f1226f74df6ab664e3a89555865b25f69b132f4bac WatchSource:0}: Error finding container 9db27b7d34ef076f21d703f1226f74df6ab664e3a89555865b25f69b132f4bac: Status 404 returned error can't find the container with id 9db27b7d34ef076f21d703f1226f74df6ab664e3a89555865b25f69b132f4bac
Jun 15 05:58:24.134279 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:24.134183    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-2wwsz" event={"ID":"9c2c0136-4539-4589-b135-969530039e2e","Type":"ContainerStarted","Data":"e47a2c397ede8bbb732aa92f9e0fbb7b091fe52230488230043f5b46a40bfacc"}
Jun 15 05:58:24.134279 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:24.134223    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-2wwsz" event={"ID":"9c2c0136-4539-4589-b135-969530039e2e","Type":"ContainerStarted","Data":"9db27b7d34ef076f21d703f1226f74df6ab664e3a89555865b25f69b132f4bac"}
Jun 15 05:58:26.145886 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:26.145852    2572 generic.go:356] "Generic (PLEG): container finished" podID="9c2c0136-4539-4589-b135-969530039e2e" containerID="e47a2c397ede8bbb732aa92f9e0fbb7b091fe52230488230043f5b46a40bfacc" exitCode=0
Jun 15 05:58:26.146263 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:26.145891    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-2wwsz" event={"ID":"9c2c0136-4539-4589-b135-969530039e2e","Type":"ContainerDied","Data":"e47a2c397ede8bbb732aa92f9e0fbb7b091fe52230488230043f5b46a40bfacc"}
Jun 15 05:58:27.152308 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:27.152272    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-2wwsz" event={"ID":"9c2c0136-4539-4589-b135-969530039e2e","Type":"ContainerStarted","Data":"f8bfd4ee63334101f96d59b9b31c20ce69cb0e226c37178d30d123bb8b1eb7c4"}
Jun 15 05:58:27.179569 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:27.179518    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-2wwsz" podStartSLOduration=2.287142298 podStartE2EDuration="4.179499683s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.892357385s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:58:27.178160142 +0000 UTC m=+622.326340139" watchObservedRunningTime="2026-06-15 05:58:27.179499683 +0000 UTC m=+622.327679680"
Jun 15 05:58:32.177127 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:32.177093    2572 generic.go:356] "Generic (PLEG): container finished" podID="9c2c0136-4539-4589-b135-969530039e2e" containerID="f8bfd4ee63334101f96d59b9b31c20ce69cb0e226c37178d30d123bb8b1eb7c4" exitCode=0
Jun 15 05:58:32.177515 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:32.177166    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-2wwsz" event={"ID":"9c2c0136-4539-4589-b135-969530039e2e","Type":"ContainerDied","Data":"f8bfd4ee63334101f96d59b9b31c20ce69cb0e226c37178d30d123bb8b1eb7c4"}
Jun 15 05:58:33.318478 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:33.318452    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-2wwsz"
Jun 15 05:58:33.491034 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:33.490933    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxv26\" (UniqueName: \"kubernetes.io/projected/9c2c0136-4539-4589-b135-969530039e2e-kube-api-access-jxv26\") pod \"9c2c0136-4539-4589-b135-969530039e2e\" (UID: \"9c2c0136-4539-4589-b135-969530039e2e\") "
Jun 15 05:58:33.491034 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:33.490975    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c2c0136-4539-4589-b135-969530039e2e-model-cache\") pod \"9c2c0136-4539-4589-b135-969530039e2e\" (UID: \"9c2c0136-4539-4589-b135-969530039e2e\") "
Jun 15 05:58:33.493043 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:33.493016    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c2c0136-4539-4589-b135-969530039e2e-kube-api-access-jxv26" (OuterVolumeSpecName: "kube-api-access-jxv26") pod "9c2c0136-4539-4589-b135-969530039e2e" (UID: "9c2c0136-4539-4589-b135-969530039e2e"). InnerVolumeSpecName "kube-api-access-jxv26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 05:58:33.513119 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:33.513086    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c2c0136-4539-4589-b135-969530039e2e-model-cache" (OuterVolumeSpecName: "model-cache") pod "9c2c0136-4539-4589-b135-969530039e2e" (UID: "9c2c0136-4539-4589-b135-969530039e2e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 05:58:33.592439 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:33.592409    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxv26\" (UniqueName: \"kubernetes.io/projected/9c2c0136-4539-4589-b135-969530039e2e-kube-api-access-jxv26\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:58:33.592439 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:33.592437    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c2c0136-4539-4589-b135-969530039e2e-model-cache\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:58:34.187352 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:34.187323    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-2wwsz"
Jun 15 05:58:34.187352 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:34.187332    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-2wwsz" event={"ID":"9c2c0136-4539-4589-b135-969530039e2e","Type":"ContainerDied","Data":"9db27b7d34ef076f21d703f1226f74df6ab664e3a89555865b25f69b132f4bac"}
Jun 15 05:58:34.187562 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:34.187398    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9db27b7d34ef076f21d703f1226f74df6ab664e3a89555865b25f69b132f4bac"
Jun 15 05:58:34.818408 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:34.818377    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-2fpfj"]
Jun 15 05:58:34.818792 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:34.818723    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c2c0136-4539-4589-b135-969530039e2e" containerName="s3-tls-init-custom"
Jun 15 05:58:34.818792 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:34.818735    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c2c0136-4539-4589-b135-969530039e2e" containerName="s3-tls-init-custom"
Jun 15 05:58:34.818792 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:34.818760    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c2c0136-4539-4589-b135-969530039e2e" containerName="download-hf-model"
Jun 15 05:58:34.818792 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:34.818765    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c2c0136-4539-4589-b135-969530039e2e" containerName="download-hf-model"
Jun 15 05:58:34.818941 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:34.818815    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c2c0136-4539-4589-b135-969530039e2e" containerName="s3-tls-init-custom"
Jun 15 05:58:34.822236 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:34.822222    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-2fpfj"
Jun 15 05:58:34.824602 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:34.824582    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-wtsnv\""
Jun 15 05:58:34.824806 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:34.824791    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jun 15 05:58:34.824880 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:34.824793    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Jun 15 05:58:34.832205 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:34.832184    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-2fpfj"]
Jun 15 05:58:34.903949 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:34.903915    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smpkj\" (UniqueName: \"kubernetes.io/projected/c3e6faac-3bd6-46b7-a063-d693523c800c-kube-api-access-smpkj\") pod \"seaweedfs-tls-serving-67b7477c6d-2fpfj\" (UID: \"c3e6faac-3bd6-46b7-a063-d693523c800c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-2fpfj"
Jun 15 05:58:34.904092 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:34.904032    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/c3e6faac-3bd6-46b7-a063-d693523c800c-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-2fpfj\" (UID: \"c3e6faac-3bd6-46b7-a063-d693523c800c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-2fpfj"
Jun 15 05:58:34.904092 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:34.904058    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c3e6faac-3bd6-46b7-a063-d693523c800c-data\") pod \"seaweedfs-tls-serving-67b7477c6d-2fpfj\" (UID: \"c3e6faac-3bd6-46b7-a063-d693523c800c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-2fpfj"
Jun 15 05:58:35.004715 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:35.004671    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/c3e6faac-3bd6-46b7-a063-d693523c800c-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-2fpfj\" (UID: \"c3e6faac-3bd6-46b7-a063-d693523c800c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-2fpfj"
Jun 15 05:58:35.004715 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:35.004725    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c3e6faac-3bd6-46b7-a063-d693523c800c-data\") pod \"seaweedfs-tls-serving-67b7477c6d-2fpfj\" (UID: \"c3e6faac-3bd6-46b7-a063-d693523c800c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-2fpfj"
Jun 15 05:58:35.004987 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:35.004750    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smpkj\" (UniqueName: \"kubernetes.io/projected/c3e6faac-3bd6-46b7-a063-d693523c800c-kube-api-access-smpkj\") pod \"seaweedfs-tls-serving-67b7477c6d-2fpfj\" (UID: \"c3e6faac-3bd6-46b7-a063-d693523c800c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-2fpfj"
Jun 15 05:58:35.004987 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:58:35.004814    2572 projected.go:266] Couldn't get secret kserve/seaweedfs-tls-serving: secret "seaweedfs-tls-serving" not found
Jun 15 05:58:35.004987 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:58:35.004830    2572 projected.go:196] Error preparing data for projected volume seaweedfs-tls-serving for pod kserve/seaweedfs-tls-serving-67b7477c6d-2fpfj: secret "seaweedfs-tls-serving" not found
Jun 15 05:58:35.004987 ip-10-0-138-16 kubenswrapper[2572]: E0615 05:58:35.004884    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c3e6faac-3bd6-46b7-a063-d693523c800c-seaweedfs-tls-serving podName:c3e6faac-3bd6-46b7-a063-d693523c800c nodeName:}" failed. No retries permitted until 2026-06-15 05:58:35.504868787 +0000 UTC m=+630.653048762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "seaweedfs-tls-serving" (UniqueName: "kubernetes.io/projected/c3e6faac-3bd6-46b7-a063-d693523c800c-seaweedfs-tls-serving") pod "seaweedfs-tls-serving-67b7477c6d-2fpfj" (UID: "c3e6faac-3bd6-46b7-a063-d693523c800c") : secret "seaweedfs-tls-serving" not found
Jun 15 05:58:35.005191 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:35.005111    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c3e6faac-3bd6-46b7-a063-d693523c800c-data\") pod \"seaweedfs-tls-serving-67b7477c6d-2fpfj\" (UID: \"c3e6faac-3bd6-46b7-a063-d693523c800c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-2fpfj"
Jun 15 05:58:35.014699 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:35.014671    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smpkj\" (UniqueName: \"kubernetes.io/projected/c3e6faac-3bd6-46b7-a063-d693523c800c-kube-api-access-smpkj\") pod \"seaweedfs-tls-serving-67b7477c6d-2fpfj\" (UID: \"c3e6faac-3bd6-46b7-a063-d693523c800c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-2fpfj"
Jun 15 05:58:35.510621 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:35.510594    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/c3e6faac-3bd6-46b7-a063-d693523c800c-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-2fpfj\" (UID: \"c3e6faac-3bd6-46b7-a063-d693523c800c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-2fpfj"
Jun 15 05:58:35.512954 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:35.512935    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/c3e6faac-3bd6-46b7-a063-d693523c800c-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-2fpfj\" (UID: \"c3e6faac-3bd6-46b7-a063-d693523c800c\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-2fpfj"
Jun 15 05:58:35.730933 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:35.730889    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-2fpfj"
Jun 15 05:58:35.862018 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:35.861991    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-2fpfj"]
Jun 15 05:58:35.864634 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:58:35.864607    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3e6faac_3bd6_46b7_a063_d693523c800c.slice/crio-1cc23e6d34eac2423ff4c41f556996deb2270bd9de6c870a4d8cb9e91d7ffa34 WatchSource:0}: Error finding container 1cc23e6d34eac2423ff4c41f556996deb2270bd9de6c870a4d8cb9e91d7ffa34: Status 404 returned error can't find the container with id 1cc23e6d34eac2423ff4c41f556996deb2270bd9de6c870a4d8cb9e91d7ffa34
Jun 15 05:58:36.197044 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:36.197010    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-2fpfj" event={"ID":"c3e6faac-3bd6-46b7-a063-d693523c800c","Type":"ContainerStarted","Data":"1cc23e6d34eac2423ff4c41f556996deb2270bd9de6c870a4d8cb9e91d7ffa34"}
Jun 15 05:58:39.212254 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:39.212220    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-2fpfj" event={"ID":"c3e6faac-3bd6-46b7-a063-d693523c800c","Type":"ContainerStarted","Data":"e26edbe010426be3ef5c49f024b730c7ccff41ac9ca4106ece274819b40671c6"}
Jun 15 05:58:39.230011 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:39.229950    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-2fpfj" podStartSLOduration=2.356928498 podStartE2EDuration="5.229931582s" totalImagesPullingTime="2.873003084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:58:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:58:39.229486813 +0000 UTC m=+634.377666811" watchObservedRunningTime="2026-06-15 05:58:39.229931582 +0000 UTC m=+634.378111582"
Jun 15 05:58:39.782441 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:39.782401    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-b5gmm"]
Jun 15 05:58:39.787252 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:39.787229    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-b5gmm"
Jun 15 05:58:39.797041 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:39.797018    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-b5gmm"]
Jun 15 05:58:39.953481 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:39.953449    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e00644c2-3afe-47da-957b-8bdc8e6f5c24-model-cache\") pod \"s3-tls-init-serving-b5gmm\" (UID: \"e00644c2-3afe-47da-957b-8bdc8e6f5c24\") " pod="kserve/s3-tls-init-serving-b5gmm"
Jun 15 05:58:39.953664 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:39.953532    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsl5c\" (UniqueName: \"kubernetes.io/projected/e00644c2-3afe-47da-957b-8bdc8e6f5c24-kube-api-access-fsl5c\") pod \"s3-tls-init-serving-b5gmm\" (UID: \"e00644c2-3afe-47da-957b-8bdc8e6f5c24\") " pod="kserve/s3-tls-init-serving-b5gmm"
Jun 15 05:58:40.054895 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:40.054805    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsl5c\" (UniqueName: \"kubernetes.io/projected/e00644c2-3afe-47da-957b-8bdc8e6f5c24-kube-api-access-fsl5c\") pod \"s3-tls-init-serving-b5gmm\" (UID: \"e00644c2-3afe-47da-957b-8bdc8e6f5c24\") " pod="kserve/s3-tls-init-serving-b5gmm"
Jun 15 05:58:40.054895 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:40.054893    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e00644c2-3afe-47da-957b-8bdc8e6f5c24-model-cache\") pod \"s3-tls-init-serving-b5gmm\" (UID: \"e00644c2-3afe-47da-957b-8bdc8e6f5c24\") " pod="kserve/s3-tls-init-serving-b5gmm"
Jun 15 05:58:40.055228 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:40.055212    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e00644c2-3afe-47da-957b-8bdc8e6f5c24-model-cache\") pod \"s3-tls-init-serving-b5gmm\" (UID: \"e00644c2-3afe-47da-957b-8bdc8e6f5c24\") " pod="kserve/s3-tls-init-serving-b5gmm"
Jun 15 05:58:40.067131 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:40.067101    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsl5c\" (UniqueName: \"kubernetes.io/projected/e00644c2-3afe-47da-957b-8bdc8e6f5c24-kube-api-access-fsl5c\") pod \"s3-tls-init-serving-b5gmm\" (UID: \"e00644c2-3afe-47da-957b-8bdc8e6f5c24\") " pod="kserve/s3-tls-init-serving-b5gmm"
Jun 15 05:58:40.105018 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:40.104984    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-b5gmm"
Jun 15 05:58:40.237041 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:40.237001    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-b5gmm"]
Jun 15 05:58:40.239332 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:58:40.239303    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode00644c2_3afe_47da_957b_8bdc8e6f5c24.slice/crio-20b47f44e42366dad39ccd230b5ca72e4f42501ad4e8739398ac2f792c80aa0e WatchSource:0}: Error finding container 20b47f44e42366dad39ccd230b5ca72e4f42501ad4e8739398ac2f792c80aa0e: Status 404 returned error can't find the container with id 20b47f44e42366dad39ccd230b5ca72e4f42501ad4e8739398ac2f792c80aa0e
Jun 15 05:58:41.226278 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:41.226048    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-b5gmm" event={"ID":"e00644c2-3afe-47da-957b-8bdc8e6f5c24","Type":"ContainerStarted","Data":"44fb7b3347b65480f489fe363836362e99aaec9664737a8369a17f65a6670507"}
Jun 15 05:58:41.226278 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:41.226115    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-b5gmm" event={"ID":"e00644c2-3afe-47da-957b-8bdc8e6f5c24","Type":"ContainerStarted","Data":"20b47f44e42366dad39ccd230b5ca72e4f42501ad4e8739398ac2f792c80aa0e"}
Jun 15 05:58:42.231503 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:42.231471    2572 generic.go:356] "Generic (PLEG): container finished" podID="e00644c2-3afe-47da-957b-8bdc8e6f5c24" containerID="44fb7b3347b65480f489fe363836362e99aaec9664737a8369a17f65a6670507" exitCode=0
Jun 15 05:58:42.231503 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:42.231505    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-b5gmm" event={"ID":"e00644c2-3afe-47da-957b-8bdc8e6f5c24","Type":"ContainerDied","Data":"44fb7b3347b65480f489fe363836362e99aaec9664737a8369a17f65a6670507"}
Jun 15 05:58:43.239822 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:43.239789    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-b5gmm" event={"ID":"e00644c2-3afe-47da-957b-8bdc8e6f5c24","Type":"ContainerStarted","Data":"f86cc6a6f144fba93759619d2dc8492dfea44c63c7c38260334a0efa5c025991"}
Jun 15 05:58:43.260340 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:43.260292    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-b5gmm" podStartSLOduration=3.064587052 podStartE2EDuration="4.260277362s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.19569031s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:58:43.259176581 +0000 UTC m=+638.407356578" watchObservedRunningTime="2026-06-15 05:58:43.260277362 +0000 UTC m=+638.408457360"
Jun 15 05:58:50.272319 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:50.272286    2572 generic.go:356] "Generic (PLEG): container finished" podID="e00644c2-3afe-47da-957b-8bdc8e6f5c24" containerID="f86cc6a6f144fba93759619d2dc8492dfea44c63c7c38260334a0efa5c025991" exitCode=0
Jun 15 05:58:50.272709 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:50.272325    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-b5gmm" event={"ID":"e00644c2-3afe-47da-957b-8bdc8e6f5c24","Type":"ContainerDied","Data":"f86cc6a6f144fba93759619d2dc8492dfea44c63c7c38260334a0efa5c025991"}
Jun 15 05:58:51.425330 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:51.425308    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-b5gmm"
Jun 15 05:58:51.451541 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:51.451515    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e00644c2-3afe-47da-957b-8bdc8e6f5c24-model-cache\") pod \"e00644c2-3afe-47da-957b-8bdc8e6f5c24\" (UID: \"e00644c2-3afe-47da-957b-8bdc8e6f5c24\") "
Jun 15 05:58:51.451686 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:51.451558    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsl5c\" (UniqueName: \"kubernetes.io/projected/e00644c2-3afe-47da-957b-8bdc8e6f5c24-kube-api-access-fsl5c\") pod \"e00644c2-3afe-47da-957b-8bdc8e6f5c24\" (UID: \"e00644c2-3afe-47da-957b-8bdc8e6f5c24\") "
Jun 15 05:58:51.453983 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:51.453943    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e00644c2-3afe-47da-957b-8bdc8e6f5c24-kube-api-access-fsl5c" (OuterVolumeSpecName: "kube-api-access-fsl5c") pod "e00644c2-3afe-47da-957b-8bdc8e6f5c24" (UID: "e00644c2-3afe-47da-957b-8bdc8e6f5c24"). InnerVolumeSpecName "kube-api-access-fsl5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 05:58:51.474333 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:51.474302    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e00644c2-3afe-47da-957b-8bdc8e6f5c24-model-cache" (OuterVolumeSpecName: "model-cache") pod "e00644c2-3afe-47da-957b-8bdc8e6f5c24" (UID: "e00644c2-3afe-47da-957b-8bdc8e6f5c24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 05:58:51.552769 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:51.552692    2572 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e00644c2-3afe-47da-957b-8bdc8e6f5c24-model-cache\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:58:51.552769 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:51.552718    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsl5c\" (UniqueName: \"kubernetes.io/projected/e00644c2-3afe-47da-957b-8bdc8e6f5c24-kube-api-access-fsl5c\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 05:58:52.281891 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:52.281852    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-b5gmm" event={"ID":"e00644c2-3afe-47da-957b-8bdc8e6f5c24","Type":"ContainerDied","Data":"20b47f44e42366dad39ccd230b5ca72e4f42501ad4e8739398ac2f792c80aa0e"}
Jun 15 05:58:52.281891 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:52.281890    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-b5gmm"
Jun 15 05:58:52.281891 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:58:52.281896    2572 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20b47f44e42366dad39ccd230b5ca72e4f42501ad4e8739398ac2f792c80aa0e"
Jun 15 05:59:03.174510 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.174475    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"]
Jun 15 05:59:03.176945 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.174844    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e00644c2-3afe-47da-957b-8bdc8e6f5c24" containerName="download-hf-model"
Jun 15 05:59:03.176945 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.174855    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e00644c2-3afe-47da-957b-8bdc8e6f5c24" containerName="download-hf-model"
Jun 15 05:59:03.176945 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.174867    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e00644c2-3afe-47da-957b-8bdc8e6f5c24" containerName="s3-tls-init-serving"
Jun 15 05:59:03.176945 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.174872    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="e00644c2-3afe-47da-957b-8bdc8e6f5c24" containerName="s3-tls-init-serving"
Jun 15 05:59:03.176945 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.174935    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="e00644c2-3afe-47da-957b-8bdc8e6f5c24" containerName="s3-tls-init-serving"
Jun 15 05:59:03.178522 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.178506    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"
Jun 15 05:59:03.181624 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.181597    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jun 15 05:59:03.181761 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.181731    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jun 15 05:59:03.181761 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.181732    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 15 05:59:03.181853 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.181807    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 15 05:59:03.181853 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.181831    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-q7vt7\""
Jun 15 05:59:03.182324 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.182306    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jun 15 05:59:03.182445 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.182345    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jun 15 05:59:03.195559 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.195531    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"]
Jun 15 05:59:03.260064 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.260027    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e023b59-ba30-435a-8e0c-5867698c9691-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-58696db7df-r9jwh\" (UID: \"6e023b59-ba30-435a-8e0c-5867698c9691\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"
Jun 15 05:59:03.260265 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.260086    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5jgl\" (UniqueName: \"kubernetes.io/projected/6e023b59-ba30-435a-8e0c-5867698c9691-kube-api-access-j5jgl\") pod \"isvc-sklearn-batcher-predictor-58696db7df-r9jwh\" (UID: \"6e023b59-ba30-435a-8e0c-5867698c9691\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"
Jun 15 05:59:03.260265 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.260121    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e023b59-ba30-435a-8e0c-5867698c9691-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-58696db7df-r9jwh\" (UID: \"6e023b59-ba30-435a-8e0c-5867698c9691\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"
Jun 15 05:59:03.260265 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.260187    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e023b59-ba30-435a-8e0c-5867698c9691-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-58696db7df-r9jwh\" (UID: \"6e023b59-ba30-435a-8e0c-5867698c9691\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"
Jun 15 05:59:03.260265 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.260236    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e023b59-ba30-435a-8e0c-5867698c9691-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-58696db7df-r9jwh\" (UID: \"6e023b59-ba30-435a-8e0c-5867698c9691\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"
Jun 15 05:59:03.360893 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.360859    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e023b59-ba30-435a-8e0c-5867698c9691-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-58696db7df-r9jwh\" (UID: \"6e023b59-ba30-435a-8e0c-5867698c9691\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"
Jun 15 05:59:03.361077 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.360909    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e023b59-ba30-435a-8e0c-5867698c9691-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-58696db7df-r9jwh\" (UID: \"6e023b59-ba30-435a-8e0c-5867698c9691\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"
Jun 15 05:59:03.361077 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.361034    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5jgl\" (UniqueName: \"kubernetes.io/projected/6e023b59-ba30-435a-8e0c-5867698c9691-kube-api-access-j5jgl\") pod \"isvc-sklearn-batcher-predictor-58696db7df-r9jwh\" (UID: \"6e023b59-ba30-435a-8e0c-5867698c9691\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"
Jun 15 05:59:03.361237 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.361086    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e023b59-ba30-435a-8e0c-5867698c9691-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-58696db7df-r9jwh\" (UID: \"6e023b59-ba30-435a-8e0c-5867698c9691\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"
Jun 15 05:59:03.361237 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.361174    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e023b59-ba30-435a-8e0c-5867698c9691-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-58696db7df-r9jwh\" (UID: \"6e023b59-ba30-435a-8e0c-5867698c9691\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"
Jun 15 05:59:03.361341 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.361291    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e023b59-ba30-435a-8e0c-5867698c9691-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-58696db7df-r9jwh\" (UID: \"6e023b59-ba30-435a-8e0c-5867698c9691\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"
Jun 15 05:59:03.361675 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.361657    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e023b59-ba30-435a-8e0c-5867698c9691-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-58696db7df-r9jwh\" (UID: \"6e023b59-ba30-435a-8e0c-5867698c9691\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"
Jun 15 05:59:03.361778 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.361760    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e023b59-ba30-435a-8e0c-5867698c9691-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-58696db7df-r9jwh\" (UID: \"6e023b59-ba30-435a-8e0c-5867698c9691\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"
Jun 15 05:59:03.363520 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.363499    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e023b59-ba30-435a-8e0c-5867698c9691-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-58696db7df-r9jwh\" (UID: \"6e023b59-ba30-435a-8e0c-5867698c9691\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"
Jun 15 05:59:03.369761 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.369739    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5jgl\" (UniqueName: \"kubernetes.io/projected/6e023b59-ba30-435a-8e0c-5867698c9691-kube-api-access-j5jgl\") pod \"isvc-sklearn-batcher-predictor-58696db7df-r9jwh\" (UID: \"6e023b59-ba30-435a-8e0c-5867698c9691\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"
Jun 15 05:59:03.488951 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.488867    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"
Jun 15 05:59:03.624951 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:03.624920    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"]
Jun 15 05:59:03.627487 ip-10-0-138-16 kubenswrapper[2572]: W0615 05:59:03.627457    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e023b59_ba30_435a_8e0c_5867698c9691.slice/crio-84e0fa012eef9ce31b8910a9b68ee8211c5fddabeef2b255cbcfff28da7d32b4 WatchSource:0}: Error finding container 84e0fa012eef9ce31b8910a9b68ee8211c5fddabeef2b255cbcfff28da7d32b4: Status 404 returned error can't find the container with id 84e0fa012eef9ce31b8910a9b68ee8211c5fddabeef2b255cbcfff28da7d32b4
Jun 15 05:59:04.351441 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:04.351403    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" event={"ID":"6e023b59-ba30-435a-8e0c-5867698c9691","Type":"ContainerStarted","Data":"233e61b6ec442eb6bdfaae02e4b073129fbe3e06fa665c4f64bd7e81a5bae6fe"}
Jun 15 05:59:04.351829 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:04.351476    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" event={"ID":"6e023b59-ba30-435a-8e0c-5867698c9691","Type":"ContainerStarted","Data":"84e0fa012eef9ce31b8910a9b68ee8211c5fddabeef2b255cbcfff28da7d32b4"}
Jun 15 05:59:08.371136 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:08.371104    2572 generic.go:356] "Generic (PLEG): container finished" podID="6e023b59-ba30-435a-8e0c-5867698c9691" containerID="233e61b6ec442eb6bdfaae02e4b073129fbe3e06fa665c4f64bd7e81a5bae6fe" exitCode=0
Jun 15 05:59:08.371521 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:08.371179    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" event={"ID":"6e023b59-ba30-435a-8e0c-5867698c9691","Type":"ContainerDied","Data":"233e61b6ec442eb6bdfaae02e4b073129fbe3e06fa665c4f64bd7e81a5bae6fe"}
Jun 15 05:59:22.457039 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:22.456935    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" event={"ID":"6e023b59-ba30-435a-8e0c-5867698c9691","Type":"ContainerStarted","Data":"2d37a3790805ba871bc4d696897d53c7ea8e4478efb57148bb6f74b290e0785c"}
Jun 15 05:59:24.468945 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:24.468911    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" event={"ID":"6e023b59-ba30-435a-8e0c-5867698c9691","Type":"ContainerStarted","Data":"24651c619a030b87190f955c4fb974512daf2443c057f4518d11b29811893aa3"}
Jun 15 05:59:27.485249 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:27.485212    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" event={"ID":"6e023b59-ba30-435a-8e0c-5867698c9691","Type":"ContainerStarted","Data":"7f7adf7de2c7599fc99cfa0ff6efda6427bda043527ef507329b0c9f5a55fdcb"}
Jun 15 05:59:27.485692 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:27.485471    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"
Jun 15 05:59:27.485692 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:27.485537    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"
Jun 15 05:59:27.485692 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:27.485575    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"
Jun 15 05:59:27.487019 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:27.486976    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jun 15 05:59:27.487731 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:27.487709    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 05:59:27.510832 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:27.510792    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podStartSLOduration=2.76862853 podStartE2EDuration="24.510777097s" totalImagesPullingTime="18.004007164s" totalInitContainerRuntime="3.738141403s" isStatefulPod=false podCreationTimestamp="2026-06-15 05:59:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 05:59:27.508108629 +0000 UTC m=+682.656288627" watchObservedRunningTime="2026-06-15 05:59:27.510777097 +0000 UTC m=+682.658957154"
Jun 15 05:59:28.489469 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:28.489424    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jun 15 05:59:28.489911 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:28.489826    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 05:59:33.493600 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:33.493571    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"
Jun 15 05:59:33.494143 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:33.494098    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jun 15 05:59:33.494653 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:33.494630    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 05:59:43.494440 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:43.494396    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jun 15 05:59:43.494846 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:43.494800    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 05:59:53.494822 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:53.494730    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jun 15 05:59:53.495196 ip-10-0-138-16 kubenswrapper[2572]: I0615 05:59:53.495173    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:00:03.494498 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:03.494457    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jun 15 06:00:03.495053 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:03.495029    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:00:13.494312 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:13.494240    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jun 15 06:00:13.494838 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:13.494720    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:00:23.494226 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:23.494180    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jun 15 06:00:23.494749 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:23.494719    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:00:33.494166 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:33.494125    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jun 15 06:00:33.494601 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:33.494577    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:00:43.495019 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:43.494991    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"
Jun 15 06:00:43.495454 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:43.495153    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"
Jun 15 06:00:58.219611 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:58.219578    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"]
Jun 15 06:00:58.220111 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:58.219837    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="kserve-container" containerID="cri-o://2d37a3790805ba871bc4d696897d53c7ea8e4478efb57148bb6f74b290e0785c" gracePeriod=30
Jun 15 06:00:58.220111 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:58.219879    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="kube-rbac-proxy" containerID="cri-o://24651c619a030b87190f955c4fb974512daf2443c057f4518d11b29811893aa3" gracePeriod=30
Jun 15 06:00:58.220111 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:58.219897    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="agent" containerID="cri-o://7f7adf7de2c7599fc99cfa0ff6efda6427bda043527ef507329b0c9f5a55fdcb" gracePeriod=30
Jun 15 06:00:58.316877 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:58.316843    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"]
Jun 15 06:00:58.319856 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:58.319837    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"
Jun 15 06:00:58.321840 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:58.321818    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jun 15 06:00:58.321840 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:58.321831    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jun 15 06:00:58.330427 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:58.330402    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"]
Jun 15 06:00:58.407760 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:58.407720    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94b7f481-6473-4126-b178-97da6b1c3d1b-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5\" (UID: \"94b7f481-6473-4126-b178-97da6b1c3d1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"
Jun 15 06:00:58.407939 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:58.407788    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94b7f481-6473-4126-b178-97da6b1c3d1b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5\" (UID: \"94b7f481-6473-4126-b178-97da6b1c3d1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"
Jun 15 06:00:58.407939 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:58.407811    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94b7f481-6473-4126-b178-97da6b1c3d1b-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5\" (UID: \"94b7f481-6473-4126-b178-97da6b1c3d1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"
Jun 15 06:00:58.407939 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:58.407844    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94b7f481-6473-4126-b178-97da6b1c3d1b-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5\" (UID: \"94b7f481-6473-4126-b178-97da6b1c3d1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"
Jun 15 06:00:58.407939 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:58.407893    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzt7b\" (UniqueName: \"kubernetes.io/projected/94b7f481-6473-4126-b178-97da6b1c3d1b-kube-api-access-jzt7b\") pod \"isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5\" (UID: \"94b7f481-6473-4126-b178-97da6b1c3d1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"
Jun 15 06:00:58.490290 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:58.490201    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jun 15 06:00:58.508893 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:58.508864    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94b7f481-6473-4126-b178-97da6b1c3d1b-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5\" (UID: \"94b7f481-6473-4126-b178-97da6b1c3d1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"
Jun 15 06:00:58.508969 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:58.508902    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzt7b\" (UniqueName: \"kubernetes.io/projected/94b7f481-6473-4126-b178-97da6b1c3d1b-kube-api-access-jzt7b\") pod \"isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5\" (UID: \"94b7f481-6473-4126-b178-97da6b1c3d1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"
Jun 15 06:00:58.509033 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:00:58.509015    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-serving-cert: secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Jun 15 06:00:58.509097 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:58.509028    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94b7f481-6473-4126-b178-97da6b1c3d1b-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5\" (UID: \"94b7f481-6473-4126-b178-97da6b1c3d1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"
Jun 15 06:00:58.509097 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:00:58.509088    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94b7f481-6473-4126-b178-97da6b1c3d1b-proxy-tls podName:94b7f481-6473-4126-b178-97da6b1c3d1b nodeName:}" failed. No retries permitted until 2026-06-15 06:00:59.009066861 +0000 UTC m=+774.157246850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/94b7f481-6473-4126-b178-97da6b1c3d1b-proxy-tls") pod "isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" (UID: "94b7f481-6473-4126-b178-97da6b1c3d1b") : secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Jun 15 06:00:58.509176 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:58.509143    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94b7f481-6473-4126-b178-97da6b1c3d1b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5\" (UID: \"94b7f481-6473-4126-b178-97da6b1c3d1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"
Jun 15 06:00:58.509228 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:58.509178    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94b7f481-6473-4126-b178-97da6b1c3d1b-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5\" (UID: \"94b7f481-6473-4126-b178-97da6b1c3d1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"
Jun 15 06:00:58.509558 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:58.509537    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94b7f481-6473-4126-b178-97da6b1c3d1b-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5\" (UID: \"94b7f481-6473-4126-b178-97da6b1c3d1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"
Jun 15 06:00:58.509740 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:58.509722    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94b7f481-6473-4126-b178-97da6b1c3d1b-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5\" (UID: \"94b7f481-6473-4126-b178-97da6b1c3d1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"
Jun 15 06:00:58.509898 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:58.509880    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94b7f481-6473-4126-b178-97da6b1c3d1b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5\" (UID: \"94b7f481-6473-4126-b178-97da6b1c3d1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"
Jun 15 06:00:58.517968 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:58.517950    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzt7b\" (UniqueName: \"kubernetes.io/projected/94b7f481-6473-4126-b178-97da6b1c3d1b-kube-api-access-jzt7b\") pod \"isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5\" (UID: \"94b7f481-6473-4126-b178-97da6b1c3d1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"
Jun 15 06:00:58.919510 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:58.919475    2572 generic.go:356] "Generic (PLEG): container finished" podID="6e023b59-ba30-435a-8e0c-5867698c9691" containerID="24651c619a030b87190f955c4fb974512daf2443c057f4518d11b29811893aa3" exitCode=2
Jun 15 06:00:58.919679 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:58.919548    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" event={"ID":"6e023b59-ba30-435a-8e0c-5867698c9691","Type":"ContainerDied","Data":"24651c619a030b87190f955c4fb974512daf2443c057f4518d11b29811893aa3"}
Jun 15 06:00:59.013219 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:59.013183    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94b7f481-6473-4126-b178-97da6b1c3d1b-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5\" (UID: \"94b7f481-6473-4126-b178-97da6b1c3d1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"
Jun 15 06:00:59.015793 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:59.015761    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94b7f481-6473-4126-b178-97da6b1c3d1b-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5\" (UID: \"94b7f481-6473-4126-b178-97da6b1c3d1b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"
Jun 15 06:00:59.231629 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:59.231546    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"
Jun 15 06:00:59.367023 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:59.366996    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"]
Jun 15 06:00:59.368908 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:00:59.368882    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b7f481_6473_4126_b178_97da6b1c3d1b.slice/crio-32d76d7de6c6015f7510e4adcc27d8293a8437c73922fb5cca1c3ba3fda1a564 WatchSource:0}: Error finding container 32d76d7de6c6015f7510e4adcc27d8293a8437c73922fb5cca1c3ba3fda1a564: Status 404 returned error can't find the container with id 32d76d7de6c6015f7510e4adcc27d8293a8437c73922fb5cca1c3ba3fda1a564
Jun 15 06:00:59.926210 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:59.926177    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" event={"ID":"94b7f481-6473-4126-b178-97da6b1c3d1b","Type":"ContainerStarted","Data":"f79315f30164ba5590e548e8c51dedeed78283673766ab9dab0f087ed89ce082"}
Jun 15 06:00:59.926420 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:00:59.926238    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" event={"ID":"94b7f481-6473-4126-b178-97da6b1c3d1b","Type":"ContainerStarted","Data":"32d76d7de6c6015f7510e4adcc27d8293a8437c73922fb5cca1c3ba3fda1a564"}
Jun 15 06:01:02.942513 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:02.942426    2572 generic.go:356] "Generic (PLEG): container finished" podID="6e023b59-ba30-435a-8e0c-5867698c9691" containerID="2d37a3790805ba871bc4d696897d53c7ea8e4478efb57148bb6f74b290e0785c" exitCode=0
Jun 15 06:01:02.942513 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:02.942474    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" event={"ID":"6e023b59-ba30-435a-8e0c-5867698c9691","Type":"ContainerDied","Data":"2d37a3790805ba871bc4d696897d53c7ea8e4478efb57148bb6f74b290e0785c"}
Jun 15 06:01:03.490128 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:03.490090    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jun 15 06:01:03.494476 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:03.494455    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jun 15 06:01:03.494895 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:03.494856    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:01:03.948400 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:03.948351    2572 generic.go:356] "Generic (PLEG): container finished" podID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerID="f79315f30164ba5590e548e8c51dedeed78283673766ab9dab0f087ed89ce082" exitCode=0
Jun 15 06:01:03.948820 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:03.948425    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" event={"ID":"94b7f481-6473-4126-b178-97da6b1c3d1b","Type":"ContainerDied","Data":"f79315f30164ba5590e548e8c51dedeed78283673766ab9dab0f087ed89ce082"}
Jun 15 06:01:04.955507 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:04.955466    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" event={"ID":"94b7f481-6473-4126-b178-97da6b1c3d1b","Type":"ContainerStarted","Data":"4579258b08a22aa3eee6fa594da39b19797f93a58185b5bcde9c8b48d5bf72f9"}
Jun 15 06:01:04.955865 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:04.955556    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" event={"ID":"94b7f481-6473-4126-b178-97da6b1c3d1b","Type":"ContainerStarted","Data":"11a7efbd5a3fa3f7c0425759f0ee564a9612089fb4fe989b0c0d859be45cd626"}
Jun 15 06:01:04.955865 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:04.955579    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" event={"ID":"94b7f481-6473-4126-b178-97da6b1c3d1b","Type":"ContainerStarted","Data":"f306d73459446d6132978ee3a92287c89620518705ca453d3c986d544ec8dfe0"}
Jun 15 06:01:04.955865 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:04.955852    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"
Jun 15 06:01:04.956022 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:04.955984    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"
Jun 15 06:01:04.957416 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:04.957391    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jun 15 06:01:04.980486 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:04.980437    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podStartSLOduration=3.159907901 podStartE2EDuration="6.980424395s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.820516494s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:00:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:01:04.977711179 +0000 UTC m=+780.125891175" watchObservedRunningTime="2026-06-15 06:01:04.980424395 +0000 UTC m=+780.128604392"
Jun 15 06:01:05.960900 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:05.960860    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jun 15 06:01:05.961298 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:05.960928    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"
Jun 15 06:01:05.962152 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:05.962126    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:01:06.965243 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:06.965209    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jun 15 06:01:06.965800 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:06.965778    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:01:08.490614 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:08.490572    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jun 15 06:01:08.490994 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:08.490687    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"
Jun 15 06:01:11.969611 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:11.969584    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"
Jun 15 06:01:11.970258 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:11.970230    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jun 15 06:01:11.970615 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:11.970586    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:01:13.489885 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:13.489845    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jun 15 06:01:13.494144 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:13.494119    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jun 15 06:01:13.494475 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:13.494455    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:01:18.489954 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:18.489855    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jun 15 06:01:21.970771 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:21.970732    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jun 15 06:01:21.971146 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:21.971113    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:01:23.489952 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:23.489908    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jun 15 06:01:23.494453 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:23.494416    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jun 15 06:01:23.494595 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:23.494559    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"
Jun 15 06:01:23.494747 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:23.494724    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:01:23.494839 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:23.494829    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"
Jun 15 06:01:28.373510 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:28.373485    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"
Jun 15 06:01:28.476267 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:28.476166    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e023b59-ba30-435a-8e0c-5867698c9691-proxy-tls\") pod \"6e023b59-ba30-435a-8e0c-5867698c9691\" (UID: \"6e023b59-ba30-435a-8e0c-5867698c9691\") "
Jun 15 06:01:28.476267 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:28.476225    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5jgl\" (UniqueName: \"kubernetes.io/projected/6e023b59-ba30-435a-8e0c-5867698c9691-kube-api-access-j5jgl\") pod \"6e023b59-ba30-435a-8e0c-5867698c9691\" (UID: \"6e023b59-ba30-435a-8e0c-5867698c9691\") "
Jun 15 06:01:28.476267 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:28.476249    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e023b59-ba30-435a-8e0c-5867698c9691-kube-rbac-proxy-sar-config\") pod \"6e023b59-ba30-435a-8e0c-5867698c9691\" (UID: \"6e023b59-ba30-435a-8e0c-5867698c9691\") "
Jun 15 06:01:28.476571 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:28.476282    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e023b59-ba30-435a-8e0c-5867698c9691-kserve-provision-location\") pod \"6e023b59-ba30-435a-8e0c-5867698c9691\" (UID: \"6e023b59-ba30-435a-8e0c-5867698c9691\") "
Jun 15 06:01:28.476571 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:28.476308    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e023b59-ba30-435a-8e0c-5867698c9691-cabundle-cert\") pod \"6e023b59-ba30-435a-8e0c-5867698c9691\" (UID: \"6e023b59-ba30-435a-8e0c-5867698c9691\") "
Jun 15 06:01:28.476668 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:28.476630    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e023b59-ba30-435a-8e0c-5867698c9691-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6e023b59-ba30-435a-8e0c-5867698c9691" (UID: "6e023b59-ba30-435a-8e0c-5867698c9691"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:01:28.476716 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:28.476675    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e023b59-ba30-435a-8e0c-5867698c9691-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6e023b59-ba30-435a-8e0c-5867698c9691" (UID: "6e023b59-ba30-435a-8e0c-5867698c9691"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:01:28.476716 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:28.476699    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e023b59-ba30-435a-8e0c-5867698c9691-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6e023b59-ba30-435a-8e0c-5867698c9691" (UID: "6e023b59-ba30-435a-8e0c-5867698c9691"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:01:28.478478 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:28.478454    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e023b59-ba30-435a-8e0c-5867698c9691-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6e023b59-ba30-435a-8e0c-5867698c9691" (UID: "6e023b59-ba30-435a-8e0c-5867698c9691"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:01:28.478592 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:28.478486    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e023b59-ba30-435a-8e0c-5867698c9691-kube-api-access-j5jgl" (OuterVolumeSpecName: "kube-api-access-j5jgl") pod "6e023b59-ba30-435a-8e0c-5867698c9691" (UID: "6e023b59-ba30-435a-8e0c-5867698c9691"). InnerVolumeSpecName "kube-api-access-j5jgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:01:28.577274 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:28.577211    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e023b59-ba30-435a-8e0c-5867698c9691-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:01:28.577274 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:28.577253    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5jgl\" (UniqueName: \"kubernetes.io/projected/6e023b59-ba30-435a-8e0c-5867698c9691-kube-api-access-j5jgl\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:01:28.577274 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:28.577269    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e023b59-ba30-435a-8e0c-5867698c9691-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:01:28.577274 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:28.577284    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6e023b59-ba30-435a-8e0c-5867698c9691-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:01:28.577551 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:28.577297    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6e023b59-ba30-435a-8e0c-5867698c9691-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:01:29.069259 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:29.069215    2572 generic.go:356] "Generic (PLEG): container finished" podID="6e023b59-ba30-435a-8e0c-5867698c9691" containerID="7f7adf7de2c7599fc99cfa0ff6efda6427bda043527ef507329b0c9f5a55fdcb" exitCode=0
Jun 15 06:01:29.069661 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:29.069285    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"
Jun 15 06:01:29.069661 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:29.069293    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" event={"ID":"6e023b59-ba30-435a-8e0c-5867698c9691","Type":"ContainerDied","Data":"7f7adf7de2c7599fc99cfa0ff6efda6427bda043527ef507329b0c9f5a55fdcb"}
Jun 15 06:01:29.069661 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:29.069383    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh" event={"ID":"6e023b59-ba30-435a-8e0c-5867698c9691","Type":"ContainerDied","Data":"84e0fa012eef9ce31b8910a9b68ee8211c5fddabeef2b255cbcfff28da7d32b4"}
Jun 15 06:01:29.069661 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:29.069416    2572 scope.go:117] "RemoveContainer" containerID="7f7adf7de2c7599fc99cfa0ff6efda6427bda043527ef507329b0c9f5a55fdcb"
Jun 15 06:01:29.077902 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:29.077883    2572 scope.go:117] "RemoveContainer" containerID="24651c619a030b87190f955c4fb974512daf2443c057f4518d11b29811893aa3"
Jun 15 06:01:29.085507 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:29.085488    2572 scope.go:117] "RemoveContainer" containerID="2d37a3790805ba871bc4d696897d53c7ea8e4478efb57148bb6f74b290e0785c"
Jun 15 06:01:29.093330 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:29.093304    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"]
Jun 15 06:01:29.095504 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:29.095482    2572 scope.go:117] "RemoveContainer" containerID="233e61b6ec442eb6bdfaae02e4b073129fbe3e06fa665c4f64bd7e81a5bae6fe"
Jun 15 06:01:29.098326 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:29.098306    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-58696db7df-r9jwh"]
Jun 15 06:01:29.109448 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:29.109425    2572 scope.go:117] "RemoveContainer" containerID="7f7adf7de2c7599fc99cfa0ff6efda6427bda043527ef507329b0c9f5a55fdcb"
Jun 15 06:01:29.109726 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:01:29.109706    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f7adf7de2c7599fc99cfa0ff6efda6427bda043527ef507329b0c9f5a55fdcb\": container with ID starting with 7f7adf7de2c7599fc99cfa0ff6efda6427bda043527ef507329b0c9f5a55fdcb not found: ID does not exist" containerID="7f7adf7de2c7599fc99cfa0ff6efda6427bda043527ef507329b0c9f5a55fdcb"
Jun 15 06:01:29.109770 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:29.109734    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f7adf7de2c7599fc99cfa0ff6efda6427bda043527ef507329b0c9f5a55fdcb"} err="failed to get container status \"7f7adf7de2c7599fc99cfa0ff6efda6427bda043527ef507329b0c9f5a55fdcb\": rpc error: code = NotFound desc = could not find container \"7f7adf7de2c7599fc99cfa0ff6efda6427bda043527ef507329b0c9f5a55fdcb\": container with ID starting with 7f7adf7de2c7599fc99cfa0ff6efda6427bda043527ef507329b0c9f5a55fdcb not found: ID does not exist"
Jun 15 06:01:29.109770 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:29.109753    2572 scope.go:117] "RemoveContainer" containerID="24651c619a030b87190f955c4fb974512daf2443c057f4518d11b29811893aa3"
Jun 15 06:01:29.110033 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:01:29.110000    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24651c619a030b87190f955c4fb974512daf2443c057f4518d11b29811893aa3\": container with ID starting with 24651c619a030b87190f955c4fb974512daf2443c057f4518d11b29811893aa3 not found: ID does not exist" containerID="24651c619a030b87190f955c4fb974512daf2443c057f4518d11b29811893aa3"
Jun 15 06:01:29.110115 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:29.110035    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24651c619a030b87190f955c4fb974512daf2443c057f4518d11b29811893aa3"} err="failed to get container status \"24651c619a030b87190f955c4fb974512daf2443c057f4518d11b29811893aa3\": rpc error: code = NotFound desc = could not find container \"24651c619a030b87190f955c4fb974512daf2443c057f4518d11b29811893aa3\": container with ID starting with 24651c619a030b87190f955c4fb974512daf2443c057f4518d11b29811893aa3 not found: ID does not exist"
Jun 15 06:01:29.110115 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:29.110052    2572 scope.go:117] "RemoveContainer" containerID="2d37a3790805ba871bc4d696897d53c7ea8e4478efb57148bb6f74b290e0785c"
Jun 15 06:01:29.110261 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:01:29.110245    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d37a3790805ba871bc4d696897d53c7ea8e4478efb57148bb6f74b290e0785c\": container with ID starting with 2d37a3790805ba871bc4d696897d53c7ea8e4478efb57148bb6f74b290e0785c not found: ID does not exist" containerID="2d37a3790805ba871bc4d696897d53c7ea8e4478efb57148bb6f74b290e0785c"
Jun 15 06:01:29.110304 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:29.110265    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d37a3790805ba871bc4d696897d53c7ea8e4478efb57148bb6f74b290e0785c"} err="failed to get container status \"2d37a3790805ba871bc4d696897d53c7ea8e4478efb57148bb6f74b290e0785c\": rpc error: code = NotFound desc = could not find container \"2d37a3790805ba871bc4d696897d53c7ea8e4478efb57148bb6f74b290e0785c\": container with ID starting with 2d37a3790805ba871bc4d696897d53c7ea8e4478efb57148bb6f74b290e0785c not found: ID does not exist"
Jun 15 06:01:29.110304 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:29.110277    2572 scope.go:117] "RemoveContainer" containerID="233e61b6ec442eb6bdfaae02e4b073129fbe3e06fa665c4f64bd7e81a5bae6fe"
Jun 15 06:01:29.110539 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:01:29.110522    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"233e61b6ec442eb6bdfaae02e4b073129fbe3e06fa665c4f64bd7e81a5bae6fe\": container with ID starting with 233e61b6ec442eb6bdfaae02e4b073129fbe3e06fa665c4f64bd7e81a5bae6fe not found: ID does not exist" containerID="233e61b6ec442eb6bdfaae02e4b073129fbe3e06fa665c4f64bd7e81a5bae6fe"
Jun 15 06:01:29.110593 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:29.110542    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"233e61b6ec442eb6bdfaae02e4b073129fbe3e06fa665c4f64bd7e81a5bae6fe"} err="failed to get container status \"233e61b6ec442eb6bdfaae02e4b073129fbe3e06fa665c4f64bd7e81a5bae6fe\": rpc error: code = NotFound desc = could not find container \"233e61b6ec442eb6bdfaae02e4b073129fbe3e06fa665c4f64bd7e81a5bae6fe\": container with ID starting with 233e61b6ec442eb6bdfaae02e4b073129fbe3e06fa665c4f64bd7e81a5bae6fe not found: ID does not exist"
Jun 15 06:01:29.326185 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:29.326109    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" path="/var/lib/kubelet/pods/6e023b59-ba30-435a-8e0c-5867698c9691/volumes"
Jun 15 06:01:31.970833 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:31.970795    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jun 15 06:01:31.971254 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:31.971227    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:01:41.970905 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:41.970430    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jun 15 06:01:41.972333 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:41.971339    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:01:51.970330 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:51.970290    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jun 15 06:01:51.970743 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:01:51.970674    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:02:01.970579 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:01.970539    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jun 15 06:02:01.971034 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:01.971011    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:02:11.970378 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:11.970324    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jun 15 06:02:11.970809 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:11.970778    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:02:17.331325 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:17.331292    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"
Jun 15 06:02:17.331792 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:17.331381    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"
Jun 15 06:02:23.505706 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:23.505675    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"]
Jun 15 06:02:23.506123 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:23.505939    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="kserve-container" containerID="cri-o://f306d73459446d6132978ee3a92287c89620518705ca453d3c986d544ec8dfe0" gracePeriod=30
Jun 15 06:02:23.506123 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:23.505978    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="agent" containerID="cri-o://4579258b08a22aa3eee6fa594da39b19797f93a58185b5bcde9c8b48d5bf72f9" gracePeriod=30
Jun 15 06:02:23.506123 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:23.506057    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="kube-rbac-proxy" containerID="cri-o://11a7efbd5a3fa3f7c0425759f0ee564a9612089fb4fe989b0c0d859be45cd626" gracePeriod=30
Jun 15 06:02:23.537423 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:23.537392    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm"]
Jun 15 06:02:23.537748 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:23.537737    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="kube-rbac-proxy"
Jun 15 06:02:23.537798 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:23.537750    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="kube-rbac-proxy"
Jun 15 06:02:23.537798 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:23.537761    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="storage-initializer"
Jun 15 06:02:23.537798 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:23.537768    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="storage-initializer"
Jun 15 06:02:23.537798 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:23.537776    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="kserve-container"
Jun 15 06:02:23.537798 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:23.537782    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="kserve-container"
Jun 15 06:02:23.537798 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:23.537788    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="agent"
Jun 15 06:02:23.537798 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:23.537793    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="agent"
Jun 15 06:02:23.538000 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:23.537855    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="kserve-container"
Jun 15 06:02:23.538000 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:23.537865    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="kube-rbac-proxy"
Jun 15 06:02:23.538000 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:23.537871    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e023b59-ba30-435a-8e0c-5867698c9691" containerName="agent"
Jun 15 06:02:23.541781 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:23.541752    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm"
Jun 15 06:02:23.546549 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:23.546529    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jun 15 06:02:23.546708 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:23.546615    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jun 15 06:02:23.556645 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:23.556622    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm"]
Jun 15 06:02:23.559375 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:23.559335    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b50eac5-7651-40b1-b5ef-803467f5b9de-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-hbpsm\" (UID: \"3b50eac5-7651-40b1-b5ef-803467f5b9de\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm"
Jun 15 06:02:23.559495 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:23.559454    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b50eac5-7651-40b1-b5ef-803467f5b9de-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-hbpsm\" (UID: \"3b50eac5-7651-40b1-b5ef-803467f5b9de\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm"
Jun 15 06:02:23.559596 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:23.559570    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc6zd\" (UniqueName: \"kubernetes.io/projected/3b50eac5-7651-40b1-b5ef-803467f5b9de-kube-api-access-lc6zd\") pod \"message-dumper-predictor-6575bcfdfd-hbpsm\" (UID: \"3b50eac5-7651-40b1-b5ef-803467f5b9de\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm"
Jun 15 06:02:23.660803 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:23.660773    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b50eac5-7651-40b1-b5ef-803467f5b9de-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-hbpsm\" (UID: \"3b50eac5-7651-40b1-b5ef-803467f5b9de\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm"
Jun 15 06:02:23.660976 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:23.660860    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc6zd\" (UniqueName: \"kubernetes.io/projected/3b50eac5-7651-40b1-b5ef-803467f5b9de-kube-api-access-lc6zd\") pod \"message-dumper-predictor-6575bcfdfd-hbpsm\" (UID: \"3b50eac5-7651-40b1-b5ef-803467f5b9de\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm"
Jun 15 06:02:23.660976 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:23.660900    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b50eac5-7651-40b1-b5ef-803467f5b9de-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-hbpsm\" (UID: \"3b50eac5-7651-40b1-b5ef-803467f5b9de\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm"
Jun 15 06:02:23.660976 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:02:23.660914    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-predictor-serving-cert: secret "message-dumper-predictor-serving-cert" not found
Jun 15 06:02:23.661155 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:02:23.660982    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b50eac5-7651-40b1-b5ef-803467f5b9de-proxy-tls podName:3b50eac5-7651-40b1-b5ef-803467f5b9de nodeName:}" failed. No retries permitted until 2026-06-15 06:02:24.160962425 +0000 UTC m=+859.309142401 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3b50eac5-7651-40b1-b5ef-803467f5b9de-proxy-tls") pod "message-dumper-predictor-6575bcfdfd-hbpsm" (UID: "3b50eac5-7651-40b1-b5ef-803467f5b9de") : secret "message-dumper-predictor-serving-cert" not found
Jun 15 06:02:23.661579 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:23.661560    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b50eac5-7651-40b1-b5ef-803467f5b9de-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-hbpsm\" (UID: \"3b50eac5-7651-40b1-b5ef-803467f5b9de\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm"
Jun 15 06:02:23.672238 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:23.672216    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc6zd\" (UniqueName: \"kubernetes.io/projected/3b50eac5-7651-40b1-b5ef-803467f5b9de-kube-api-access-lc6zd\") pod \"message-dumper-predictor-6575bcfdfd-hbpsm\" (UID: \"3b50eac5-7651-40b1-b5ef-803467f5b9de\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm"
Jun 15 06:02:24.165393 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:24.165341    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b50eac5-7651-40b1-b5ef-803467f5b9de-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-hbpsm\" (UID: \"3b50eac5-7651-40b1-b5ef-803467f5b9de\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm"
Jun 15 06:02:24.167739 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:24.167721    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b50eac5-7651-40b1-b5ef-803467f5b9de-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-hbpsm\" (UID: \"3b50eac5-7651-40b1-b5ef-803467f5b9de\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm"
Jun 15 06:02:24.331267 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:24.331237    2572 generic.go:356] "Generic (PLEG): container finished" podID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerID="11a7efbd5a3fa3f7c0425759f0ee564a9612089fb4fe989b0c0d859be45cd626" exitCode=2
Jun 15 06:02:24.331445 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:24.331296    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" event={"ID":"94b7f481-6473-4126-b178-97da6b1c3d1b","Type":"ContainerDied","Data":"11a7efbd5a3fa3f7c0425759f0ee564a9612089fb4fe989b0c0d859be45cd626"}
Jun 15 06:02:24.453309 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:24.453231    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm"
Jun 15 06:02:24.584811 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:24.584785    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm"]
Jun 15 06:02:24.586749 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:02:24.586723    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b50eac5_7651_40b1_b5ef_803467f5b9de.slice/crio-e26448e8de804195ef3c9205cfd4de6c220cf80fbb606b176bb944400d19336c WatchSource:0}: Error finding container e26448e8de804195ef3c9205cfd4de6c220cf80fbb606b176bb944400d19336c: Status 404 returned error can't find the container with id e26448e8de804195ef3c9205cfd4de6c220cf80fbb606b176bb944400d19336c
Jun 15 06:02:24.588720 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:24.588702    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 15 06:02:25.337202 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:25.337171    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm" event={"ID":"3b50eac5-7651-40b1-b5ef-803467f5b9de","Type":"ContainerStarted","Data":"e26448e8de804195ef3c9205cfd4de6c220cf80fbb606b176bb944400d19336c"}
Jun 15 06:02:26.346067 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:26.345968    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm" event={"ID":"3b50eac5-7651-40b1-b5ef-803467f5b9de","Type":"ContainerStarted","Data":"2cbd9fca427ef3bca2b01fce3fc8cc39287b86eba2039c8e9c2f919c4f8c5b48"}
Jun 15 06:02:26.346067 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:26.346038    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm" event={"ID":"3b50eac5-7651-40b1-b5ef-803467f5b9de","Type":"ContainerStarted","Data":"7814e7bcc864235ce46ad30f37241d19d77971082784a0289b5a7533f7877f0f"}
Jun 15 06:02:26.346491 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:26.346076    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm"
Jun 15 06:02:26.346491 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:26.346104    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm"
Jun 15 06:02:26.348032 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:26.348007    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm"
Jun 15 06:02:26.366802 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:26.366758    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm" podStartSLOduration=1.952380443 podStartE2EDuration="3.366744023s" totalImagesPullingTime="1.41436358s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:02:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:02:26.365692783 +0000 UTC m=+861.513872780" watchObservedRunningTime="2026-06-15 06:02:26.366744023 +0000 UTC m=+861.514924020"
Jun 15 06:02:26.965949 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:26.965912    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jun 15 06:02:27.319020 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:27.318937    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jun 15 06:02:27.319328 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:27.319306    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:02:28.357568 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:28.357536    2572 generic.go:356] "Generic (PLEG): container finished" podID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerID="f306d73459446d6132978ee3a92287c89620518705ca453d3c986d544ec8dfe0" exitCode=0
Jun 15 06:02:28.357928 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:28.357605    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" event={"ID":"94b7f481-6473-4126-b178-97da6b1c3d1b","Type":"ContainerDied","Data":"f306d73459446d6132978ee3a92287c89620518705ca453d3c986d544ec8dfe0"}
Jun 15 06:02:31.966243 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:31.966203    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jun 15 06:02:33.362775 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:33.362747    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm"
Jun 15 06:02:36.966172 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:36.966131    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jun 15 06:02:36.966574 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:36.966295    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"
Jun 15 06:02:37.318854 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:37.318768    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jun 15 06:02:37.319176 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:37.319144    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:02:41.965908 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:41.965870    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jun 15 06:02:43.551808 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:43.551774    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"]
Jun 15 06:02:43.555544 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:43.555529    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"
Jun 15 06:02:43.557900 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:43.557875    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jun 15 06:02:43.557900 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:43.557887    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jun 15 06:02:43.567376 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:43.567338    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"]
Jun 15 06:02:43.631777 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:43.631741    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl6tz\" (UniqueName: \"kubernetes.io/projected/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-kube-api-access-xl6tz\") pod \"isvc-logger-predictor-654b69f66-snppp\" (UID: \"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"
Jun 15 06:02:43.631777 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:43.631781    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-cabundle-cert\") pod \"isvc-logger-predictor-654b69f66-snppp\" (UID: \"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"
Jun 15 06:02:43.631984 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:43.631813    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-kserve-provision-location\") pod \"isvc-logger-predictor-654b69f66-snppp\" (UID: \"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"
Jun 15 06:02:43.631984 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:43.631846    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-654b69f66-snppp\" (UID: \"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"
Jun 15 06:02:43.631984 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:43.631872    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-proxy-tls\") pod \"isvc-logger-predictor-654b69f66-snppp\" (UID: \"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"
Jun 15 06:02:43.732846 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:43.732818    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-cabundle-cert\") pod \"isvc-logger-predictor-654b69f66-snppp\" (UID: \"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"
Jun 15 06:02:43.733030 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:43.732857    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-kserve-provision-location\") pod \"isvc-logger-predictor-654b69f66-snppp\" (UID: \"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"
Jun 15 06:02:43.733030 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:43.732884    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-654b69f66-snppp\" (UID: \"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"
Jun 15 06:02:43.733030 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:43.732913    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-proxy-tls\") pod \"isvc-logger-predictor-654b69f66-snppp\" (UID: \"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"
Jun 15 06:02:43.733030 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:43.733001    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl6tz\" (UniqueName: \"kubernetes.io/projected/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-kube-api-access-xl6tz\") pod \"isvc-logger-predictor-654b69f66-snppp\" (UID: \"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"
Jun 15 06:02:43.733584 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:02:43.733085    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-predictor-serving-cert: secret "isvc-logger-predictor-serving-cert" not found
Jun 15 06:02:43.733584 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:02:43.733151    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-proxy-tls podName:211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1 nodeName:}" failed. No retries permitted until 2026-06-15 06:02:44.233131035 +0000 UTC m=+879.381311015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-proxy-tls") pod "isvc-logger-predictor-654b69f66-snppp" (UID: "211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1") : secret "isvc-logger-predictor-serving-cert" not found
Jun 15 06:02:43.733584 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:43.733263    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-kserve-provision-location\") pod \"isvc-logger-predictor-654b69f66-snppp\" (UID: \"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"
Jun 15 06:02:43.733768 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:43.733633    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-654b69f66-snppp\" (UID: \"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"
Jun 15 06:02:43.733768 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:43.733690    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-cabundle-cert\") pod \"isvc-logger-predictor-654b69f66-snppp\" (UID: \"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"
Jun 15 06:02:43.743141 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:43.743117    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl6tz\" (UniqueName: \"kubernetes.io/projected/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-kube-api-access-xl6tz\") pod \"isvc-logger-predictor-654b69f66-snppp\" (UID: \"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"
Jun 15 06:02:44.238019 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:44.237982    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-proxy-tls\") pod \"isvc-logger-predictor-654b69f66-snppp\" (UID: \"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"
Jun 15 06:02:44.240462 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:44.240439    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-proxy-tls\") pod \"isvc-logger-predictor-654b69f66-snppp\" (UID: \"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"
Jun 15 06:02:44.494135 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:44.494022    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"
Jun 15 06:02:44.631622 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:44.631594    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"]
Jun 15 06:02:44.633613 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:02:44.633588    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod211f83c0_34ef_4cb3_8ae7_cf2c6a9889e1.slice/crio-5c69ea630217ca37df49651d187345780243bafd10cb072feca08cd324c66fbe WatchSource:0}: Error finding container 5c69ea630217ca37df49651d187345780243bafd10cb072feca08cd324c66fbe: Status 404 returned error can't find the container with id 5c69ea630217ca37df49651d187345780243bafd10cb072feca08cd324c66fbe
Jun 15 06:02:45.439635 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:45.439552    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" event={"ID":"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1","Type":"ContainerStarted","Data":"2c815aac7a17570637fec2292b578bda64471fa09e92cb193a1bc7899d1046af"}
Jun 15 06:02:45.439635 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:45.439616    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" event={"ID":"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1","Type":"ContainerStarted","Data":"5c69ea630217ca37df49651d187345780243bafd10cb072feca08cd324c66fbe"}
Jun 15 06:02:46.965950 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:46.965901    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jun 15 06:02:47.319176 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:47.319069    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:5000: connect: connection refused"
Jun 15 06:02:47.319492 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:47.319470    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:02:47.324437 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:47.324414    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"
Jun 15 06:02:47.324571 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:47.324524    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"
Jun 15 06:02:49.460898 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:49.460862    2572 generic.go:356] "Generic (PLEG): container finished" podID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerID="2c815aac7a17570637fec2292b578bda64471fa09e92cb193a1bc7899d1046af" exitCode=0
Jun 15 06:02:49.461262 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:49.460933    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" event={"ID":"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1","Type":"ContainerDied","Data":"2c815aac7a17570637fec2292b578bda64471fa09e92cb193a1bc7899d1046af"}
Jun 15 06:02:50.468377 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:50.468331    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" event={"ID":"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1","Type":"ContainerStarted","Data":"b45759f69f7b3372b3dfe909adc59e538ed52b526f41377fe7babecc45579dec"}
Jun 15 06:02:50.468764 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:50.468407    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" event={"ID":"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1","Type":"ContainerStarted","Data":"fe10bb79c344545fdf56a4ee3467eecec9b7fceba89e0335300876067a547e45"}
Jun 15 06:02:50.468764 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:50.468431    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" event={"ID":"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1","Type":"ContainerStarted","Data":"49b7780bb21db07b3c7f40013bae09841cc4c305d6a9b154c36147d891686c12"}
Jun 15 06:02:50.468764 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:50.468735    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"
Jun 15 06:02:50.468947 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:50.468920    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"
Jun 15 06:02:50.470313 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:50.470287    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jun 15 06:02:50.490883 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:50.490833    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podStartSLOduration=3.723268065 podStartE2EDuration="7.49081867s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.767550605s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:02:50.488225797 +0000 UTC m=+885.636405794" watchObservedRunningTime="2026-06-15 06:02:50.49081867 +0000 UTC m=+885.638998666"
Jun 15 06:02:51.472872 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:51.472836    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"
Jun 15 06:02:51.473349 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:51.472981    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jun 15 06:02:51.474108 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:51.474084    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:02:51.965607 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:51.965569    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jun 15 06:02:52.477775 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:52.477730    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jun 15 06:02:52.478198 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:52.478100    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:02:53.655935 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:53.655905    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"
Jun 15 06:02:53.830314 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:53.830222    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94b7f481-6473-4126-b178-97da6b1c3d1b-cabundle-cert\") pod \"94b7f481-6473-4126-b178-97da6b1c3d1b\" (UID: \"94b7f481-6473-4126-b178-97da6b1c3d1b\") "
Jun 15 06:02:53.830314 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:53.830271    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzt7b\" (UniqueName: \"kubernetes.io/projected/94b7f481-6473-4126-b178-97da6b1c3d1b-kube-api-access-jzt7b\") pod \"94b7f481-6473-4126-b178-97da6b1c3d1b\" (UID: \"94b7f481-6473-4126-b178-97da6b1c3d1b\") "
Jun 15 06:02:53.830314 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:53.830300    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94b7f481-6473-4126-b178-97da6b1c3d1b-kube-rbac-proxy-sar-config\") pod \"94b7f481-6473-4126-b178-97da6b1c3d1b\" (UID: \"94b7f481-6473-4126-b178-97da6b1c3d1b\") "
Jun 15 06:02:53.830622 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:53.830354    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94b7f481-6473-4126-b178-97da6b1c3d1b-proxy-tls\") pod \"94b7f481-6473-4126-b178-97da6b1c3d1b\" (UID: \"94b7f481-6473-4126-b178-97da6b1c3d1b\") "
Jun 15 06:02:53.830622 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:53.830403    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94b7f481-6473-4126-b178-97da6b1c3d1b-kserve-provision-location\") pod \"94b7f481-6473-4126-b178-97da6b1c3d1b\" (UID: \"94b7f481-6473-4126-b178-97da6b1c3d1b\") "
Jun 15 06:02:53.830732 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:53.830690    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b7f481-6473-4126-b178-97da6b1c3d1b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "94b7f481-6473-4126-b178-97da6b1c3d1b" (UID: "94b7f481-6473-4126-b178-97da6b1c3d1b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:02:53.830789 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:53.830733    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b7f481-6473-4126-b178-97da6b1c3d1b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "94b7f481-6473-4126-b178-97da6b1c3d1b" (UID: "94b7f481-6473-4126-b178-97da6b1c3d1b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:02:53.830847 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:53.830782    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b7f481-6473-4126-b178-97da6b1c3d1b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "94b7f481-6473-4126-b178-97da6b1c3d1b" (UID: "94b7f481-6473-4126-b178-97da6b1c3d1b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:02:53.832465 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:53.832433    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94b7f481-6473-4126-b178-97da6b1c3d1b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "94b7f481-6473-4126-b178-97da6b1c3d1b" (UID: "94b7f481-6473-4126-b178-97da6b1c3d1b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:02:53.832579 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:53.832508    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b7f481-6473-4126-b178-97da6b1c3d1b-kube-api-access-jzt7b" (OuterVolumeSpecName: "kube-api-access-jzt7b") pod "94b7f481-6473-4126-b178-97da6b1c3d1b" (UID: "94b7f481-6473-4126-b178-97da6b1c3d1b"). InnerVolumeSpecName "kube-api-access-jzt7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:02:53.931481 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:53.931453    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzt7b\" (UniqueName: \"kubernetes.io/projected/94b7f481-6473-4126-b178-97da6b1c3d1b-kube-api-access-jzt7b\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:02:53.931481 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:53.931478    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94b7f481-6473-4126-b178-97da6b1c3d1b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:02:53.931481 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:53.931487    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94b7f481-6473-4126-b178-97da6b1c3d1b-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:02:53.931696 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:53.931497    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/94b7f481-6473-4126-b178-97da6b1c3d1b-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:02:53.931696 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:53.931505    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/94b7f481-6473-4126-b178-97da6b1c3d1b-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:02:54.489229 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:54.489196    2572 generic.go:356] "Generic (PLEG): container finished" podID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerID="4579258b08a22aa3eee6fa594da39b19797f93a58185b5bcde9c8b48d5bf72f9" exitCode=0
Jun 15 06:02:54.489419 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:54.489253    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"
Jun 15 06:02:54.489419 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:54.489265    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" event={"ID":"94b7f481-6473-4126-b178-97da6b1c3d1b","Type":"ContainerDied","Data":"4579258b08a22aa3eee6fa594da39b19797f93a58185b5bcde9c8b48d5bf72f9"}
Jun 15 06:02:54.489419 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:54.489324    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5" event={"ID":"94b7f481-6473-4126-b178-97da6b1c3d1b","Type":"ContainerDied","Data":"32d76d7de6c6015f7510e4adcc27d8293a8437c73922fb5cca1c3ba3fda1a564"}
Jun 15 06:02:54.489419 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:54.489378    2572 scope.go:117] "RemoveContainer" containerID="4579258b08a22aa3eee6fa594da39b19797f93a58185b5bcde9c8b48d5bf72f9"
Jun 15 06:02:54.498211 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:54.498191    2572 scope.go:117] "RemoveContainer" containerID="11a7efbd5a3fa3f7c0425759f0ee564a9612089fb4fe989b0c0d859be45cd626"
Jun 15 06:02:54.505476 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:54.505456    2572 scope.go:117] "RemoveContainer" containerID="f306d73459446d6132978ee3a92287c89620518705ca453d3c986d544ec8dfe0"
Jun 15 06:02:54.512389 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:54.512342    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"]
Jun 15 06:02:54.513238 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:54.513139    2572 scope.go:117] "RemoveContainer" containerID="f79315f30164ba5590e548e8c51dedeed78283673766ab9dab0f087ed89ce082"
Jun 15 06:02:54.518265 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:54.518242    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-9d95f7dcb-llfv5"]
Jun 15 06:02:54.521115 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:54.521097    2572 scope.go:117] "RemoveContainer" containerID="4579258b08a22aa3eee6fa594da39b19797f93a58185b5bcde9c8b48d5bf72f9"
Jun 15 06:02:54.521395 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:02:54.521376    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4579258b08a22aa3eee6fa594da39b19797f93a58185b5bcde9c8b48d5bf72f9\": container with ID starting with 4579258b08a22aa3eee6fa594da39b19797f93a58185b5bcde9c8b48d5bf72f9 not found: ID does not exist" containerID="4579258b08a22aa3eee6fa594da39b19797f93a58185b5bcde9c8b48d5bf72f9"
Jun 15 06:02:54.521451 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:54.521405    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4579258b08a22aa3eee6fa594da39b19797f93a58185b5bcde9c8b48d5bf72f9"} err="failed to get container status \"4579258b08a22aa3eee6fa594da39b19797f93a58185b5bcde9c8b48d5bf72f9\": rpc error: code = NotFound desc = could not find container \"4579258b08a22aa3eee6fa594da39b19797f93a58185b5bcde9c8b48d5bf72f9\": container with ID starting with 4579258b08a22aa3eee6fa594da39b19797f93a58185b5bcde9c8b48d5bf72f9 not found: ID does not exist"
Jun 15 06:02:54.521451 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:54.521424    2572 scope.go:117] "RemoveContainer" containerID="11a7efbd5a3fa3f7c0425759f0ee564a9612089fb4fe989b0c0d859be45cd626"
Jun 15 06:02:54.521680 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:02:54.521662    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11a7efbd5a3fa3f7c0425759f0ee564a9612089fb4fe989b0c0d859be45cd626\": container with ID starting with 11a7efbd5a3fa3f7c0425759f0ee564a9612089fb4fe989b0c0d859be45cd626 not found: ID does not exist" containerID="11a7efbd5a3fa3f7c0425759f0ee564a9612089fb4fe989b0c0d859be45cd626"
Jun 15 06:02:54.521727 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:54.521687    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11a7efbd5a3fa3f7c0425759f0ee564a9612089fb4fe989b0c0d859be45cd626"} err="failed to get container status \"11a7efbd5a3fa3f7c0425759f0ee564a9612089fb4fe989b0c0d859be45cd626\": rpc error: code = NotFound desc = could not find container \"11a7efbd5a3fa3f7c0425759f0ee564a9612089fb4fe989b0c0d859be45cd626\": container with ID starting with 11a7efbd5a3fa3f7c0425759f0ee564a9612089fb4fe989b0c0d859be45cd626 not found: ID does not exist"
Jun 15 06:02:54.521727 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:54.521704    2572 scope.go:117] "RemoveContainer" containerID="f306d73459446d6132978ee3a92287c89620518705ca453d3c986d544ec8dfe0"
Jun 15 06:02:54.521956 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:02:54.521940    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f306d73459446d6132978ee3a92287c89620518705ca453d3c986d544ec8dfe0\": container with ID starting with f306d73459446d6132978ee3a92287c89620518705ca453d3c986d544ec8dfe0 not found: ID does not exist" containerID="f306d73459446d6132978ee3a92287c89620518705ca453d3c986d544ec8dfe0"
Jun 15 06:02:54.521998 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:54.521961    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f306d73459446d6132978ee3a92287c89620518705ca453d3c986d544ec8dfe0"} err="failed to get container status \"f306d73459446d6132978ee3a92287c89620518705ca453d3c986d544ec8dfe0\": rpc error: code = NotFound desc = could not find container \"f306d73459446d6132978ee3a92287c89620518705ca453d3c986d544ec8dfe0\": container with ID starting with f306d73459446d6132978ee3a92287c89620518705ca453d3c986d544ec8dfe0 not found: ID does not exist"
Jun 15 06:02:54.521998 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:54.521974    2572 scope.go:117] "RemoveContainer" containerID="f79315f30164ba5590e548e8c51dedeed78283673766ab9dab0f087ed89ce082"
Jun 15 06:02:54.522206 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:02:54.522189    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f79315f30164ba5590e548e8c51dedeed78283673766ab9dab0f087ed89ce082\": container with ID starting with f79315f30164ba5590e548e8c51dedeed78283673766ab9dab0f087ed89ce082 not found: ID does not exist" containerID="f79315f30164ba5590e548e8c51dedeed78283673766ab9dab0f087ed89ce082"
Jun 15 06:02:54.522242 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:54.522210    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f79315f30164ba5590e548e8c51dedeed78283673766ab9dab0f087ed89ce082"} err="failed to get container status \"f79315f30164ba5590e548e8c51dedeed78283673766ab9dab0f087ed89ce082\": rpc error: code = NotFound desc = could not find container \"f79315f30164ba5590e548e8c51dedeed78283673766ab9dab0f087ed89ce082\": container with ID starting with f79315f30164ba5590e548e8c51dedeed78283673766ab9dab0f087ed89ce082 not found: ID does not exist"
Jun 15 06:02:55.324000 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:55.323965    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" path="/var/lib/kubelet/pods/94b7f481-6473-4126-b178-97da6b1c3d1b/volumes"
Jun 15 06:02:57.481790 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:57.481762    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"
Jun 15 06:02:57.482409 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:57.482381    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jun 15 06:02:57.482704 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:02:57.482670    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:03:05.331709 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:03:05.331681    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 06:03:05.335141 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:03:05.335119    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 06:03:07.482900 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:03:07.482860    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jun 15 06:03:07.483419 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:03:07.483390    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:03:17.482328 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:03:17.482277    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jun 15 06:03:17.482769 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:03:17.482715    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:03:27.482893 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:03:27.482850    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jun 15 06:03:27.483395 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:03:27.483373    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:03:37.482308 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:03:37.482263    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jun 15 06:03:37.482880 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:03:37.482765    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:03:47.482905 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:03:47.482861    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jun 15 06:03:47.483330 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:03:47.483240    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:03:57.482343 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:03:57.482301    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jun 15 06:03:57.482881 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:03:57.482855    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:04:07.483014 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:07.482979    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"
Jun 15 06:04:07.485337 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:07.483551    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"
Jun 15 06:04:18.564502 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:18.564474    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-hbpsm_3b50eac5-7651-40b1-b5ef-803467f5b9de/kserve-container/0.log"
Jun 15 06:04:18.773638 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:18.773604    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"]
Jun 15 06:04:18.773947 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:18.773919    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="kserve-container" containerID="cri-o://49b7780bb21db07b3c7f40013bae09841cc4c305d6a9b154c36147d891686c12" gracePeriod=30
Jun 15 06:04:18.774192 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:18.773999    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="agent" containerID="cri-o://b45759f69f7b3372b3dfe909adc59e538ed52b526f41377fe7babecc45579dec" gracePeriod=30
Jun 15 06:04:18.774192 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:18.773993    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="kube-rbac-proxy" containerID="cri-o://fe10bb79c344545fdf56a4ee3467eecec9b7fceba89e0335300876067a547e45" gracePeriod=30
Jun 15 06:04:18.842529 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:18.842450    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv"]
Jun 15 06:04:18.842923 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:18.842905    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="kube-rbac-proxy"
Jun 15 06:04:18.842997 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:18.842925    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="kube-rbac-proxy"
Jun 15 06:04:18.842997 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:18.842948    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="agent"
Jun 15 06:04:18.842997 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:18.842955    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="agent"
Jun 15 06:04:18.842997 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:18.842964    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="kserve-container"
Jun 15 06:04:18.842997 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:18.842973    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="kserve-container"
Jun 15 06:04:18.842997 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:18.842984    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="storage-initializer"
Jun 15 06:04:18.842997 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:18.842989    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="storage-initializer"
Jun 15 06:04:18.843371 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:18.843066    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="kube-rbac-proxy"
Jun 15 06:04:18.843371 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:18.843075    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="agent"
Jun 15 06:04:18.843371 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:18.843088    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="94b7f481-6473-4126-b178-97da6b1c3d1b" containerName="kserve-container"
Jun 15 06:04:18.846258 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:18.846240    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv"
Jun 15 06:04:18.848211 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:18.848193    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Jun 15 06:04:18.848311 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:18.848295    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Jun 15 06:04:18.855408 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:18.855387    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv"]
Jun 15 06:04:18.887938 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:18.887903    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm"]
Jun 15 06:04:18.888479 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:18.888293    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm" podUID="3b50eac5-7651-40b1-b5ef-803467f5b9de" containerName="kserve-container" containerID="cri-o://7814e7bcc864235ce46ad30f37241d19d77971082784a0289b5a7533f7877f0f" gracePeriod=30
Jun 15 06:04:18.888479 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:18.888332    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm" podUID="3b50eac5-7651-40b1-b5ef-803467f5b9de" containerName="kube-rbac-proxy" containerID="cri-o://2cbd9fca427ef3bca2b01fce3fc8cc39287b86eba2039c8e9c2f919c4f8c5b48" gracePeriod=30
Jun 15 06:04:18.895777 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:18.895753    2572 generic.go:356] "Generic (PLEG): container finished" podID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerID="fe10bb79c344545fdf56a4ee3467eecec9b7fceba89e0335300876067a547e45" exitCode=2
Jun 15 06:04:18.895866 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:18.895827    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" event={"ID":"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1","Type":"ContainerDied","Data":"fe10bb79c344545fdf56a4ee3467eecec9b7fceba89e0335300876067a547e45"}
Jun 15 06:04:19.004988 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.004965    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-nr8lv\" (UID: \"45cc701b-3e04-4c1e-94b8-e8ea767f96d7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv"
Jun 15 06:04:19.005061 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.005017    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-nr8lv\" (UID: \"45cc701b-3e04-4c1e-94b8-e8ea767f96d7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv"
Jun 15 06:04:19.005104 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.005087    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-nr8lv\" (UID: \"45cc701b-3e04-4c1e-94b8-e8ea767f96d7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv"
Jun 15 06:04:19.005202 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.005184    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-nr8lv\" (UID: \"45cc701b-3e04-4c1e-94b8-e8ea767f96d7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv"
Jun 15 06:04:19.005242 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.005215    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njbwg\" (UniqueName: \"kubernetes.io/projected/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-kube-api-access-njbwg\") pod \"isvc-lightgbm-predictor-54f85ddf88-nr8lv\" (UID: \"45cc701b-3e04-4c1e-94b8-e8ea767f96d7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv"
Jun 15 06:04:19.106519 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.106451    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njbwg\" (UniqueName: \"kubernetes.io/projected/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-kube-api-access-njbwg\") pod \"isvc-lightgbm-predictor-54f85ddf88-nr8lv\" (UID: \"45cc701b-3e04-4c1e-94b8-e8ea767f96d7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv"
Jun 15 06:04:19.106519 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.106487    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-nr8lv\" (UID: \"45cc701b-3e04-4c1e-94b8-e8ea767f96d7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv"
Jun 15 06:04:19.106728 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.106551    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-nr8lv\" (UID: \"45cc701b-3e04-4c1e-94b8-e8ea767f96d7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv"
Jun 15 06:04:19.106728 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.106598    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-nr8lv\" (UID: \"45cc701b-3e04-4c1e-94b8-e8ea767f96d7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv"
Jun 15 06:04:19.106728 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.106686    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-nr8lv\" (UID: \"45cc701b-3e04-4c1e-94b8-e8ea767f96d7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv"
Jun 15 06:04:19.107037 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.107013    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-nr8lv\" (UID: \"45cc701b-3e04-4c1e-94b8-e8ea767f96d7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv"
Jun 15 06:04:19.107282 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.107256    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-nr8lv\" (UID: \"45cc701b-3e04-4c1e-94b8-e8ea767f96d7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv"
Jun 15 06:04:19.107282 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.107266    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-nr8lv\" (UID: \"45cc701b-3e04-4c1e-94b8-e8ea767f96d7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv"
Jun 15 06:04:19.109112 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.109091    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-nr8lv\" (UID: \"45cc701b-3e04-4c1e-94b8-e8ea767f96d7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv"
Jun 15 06:04:19.114979 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.114942    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njbwg\" (UniqueName: \"kubernetes.io/projected/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-kube-api-access-njbwg\") pod \"isvc-lightgbm-predictor-54f85ddf88-nr8lv\" (UID: \"45cc701b-3e04-4c1e-94b8-e8ea767f96d7\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv"
Jun 15 06:04:19.130638 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.130619    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm"
Jun 15 06:04:19.157582 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.157551    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv"
Jun 15 06:04:19.286669 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.286645    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv"]
Jun 15 06:04:19.288867 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:04:19.288841    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45cc701b_3e04_4c1e_94b8_e8ea767f96d7.slice/crio-81fda7e551bf458a87c6d865187339194042b652ed7b2d58e344e52eb30fb452 WatchSource:0}: Error finding container 81fda7e551bf458a87c6d865187339194042b652ed7b2d58e344e52eb30fb452: Status 404 returned error can't find the container with id 81fda7e551bf458a87c6d865187339194042b652ed7b2d58e344e52eb30fb452
Jun 15 06:04:19.308113 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.308090    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b50eac5-7651-40b1-b5ef-803467f5b9de-kube-rbac-proxy-sar-config\") pod \"3b50eac5-7651-40b1-b5ef-803467f5b9de\" (UID: \"3b50eac5-7651-40b1-b5ef-803467f5b9de\") "
Jun 15 06:04:19.308206 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.308189    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b50eac5-7651-40b1-b5ef-803467f5b9de-proxy-tls\") pod \"3b50eac5-7651-40b1-b5ef-803467f5b9de\" (UID: \"3b50eac5-7651-40b1-b5ef-803467f5b9de\") "
Jun 15 06:04:19.308254 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.308212    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lc6zd\" (UniqueName: \"kubernetes.io/projected/3b50eac5-7651-40b1-b5ef-803467f5b9de-kube-api-access-lc6zd\") pod \"3b50eac5-7651-40b1-b5ef-803467f5b9de\" (UID: \"3b50eac5-7651-40b1-b5ef-803467f5b9de\") "
Jun 15 06:04:19.308488 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.308459    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b50eac5-7651-40b1-b5ef-803467f5b9de-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3b50eac5-7651-40b1-b5ef-803467f5b9de" (UID: "3b50eac5-7651-40b1-b5ef-803467f5b9de"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:04:19.310154 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.310134    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b50eac5-7651-40b1-b5ef-803467f5b9de-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3b50eac5-7651-40b1-b5ef-803467f5b9de" (UID: "3b50eac5-7651-40b1-b5ef-803467f5b9de"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:04:19.310216 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.310161    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b50eac5-7651-40b1-b5ef-803467f5b9de-kube-api-access-lc6zd" (OuterVolumeSpecName: "kube-api-access-lc6zd") pod "3b50eac5-7651-40b1-b5ef-803467f5b9de" (UID: "3b50eac5-7651-40b1-b5ef-803467f5b9de"). InnerVolumeSpecName "kube-api-access-lc6zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:04:19.409239 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.409209    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3b50eac5-7651-40b1-b5ef-803467f5b9de-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:04:19.409239 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.409237    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3b50eac5-7651-40b1-b5ef-803467f5b9de-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:04:19.409462 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.409248    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc6zd\" (UniqueName: \"kubernetes.io/projected/3b50eac5-7651-40b1-b5ef-803467f5b9de-kube-api-access-lc6zd\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:04:19.902564 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.902525    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv" event={"ID":"45cc701b-3e04-4c1e-94b8-e8ea767f96d7","Type":"ContainerStarted","Data":"eced31ebab941fa3cc681148bf4e1e12f048c283dd8bafe373548e2d34337d40"}
Jun 15 06:04:19.903009 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.902600    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv" event={"ID":"45cc701b-3e04-4c1e-94b8-e8ea767f96d7","Type":"ContainerStarted","Data":"81fda7e551bf458a87c6d865187339194042b652ed7b2d58e344e52eb30fb452"}
Jun 15 06:04:19.904179 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.904154    2572 generic.go:356] "Generic (PLEG): container finished" podID="3b50eac5-7651-40b1-b5ef-803467f5b9de" containerID="2cbd9fca427ef3bca2b01fce3fc8cc39287b86eba2039c8e9c2f919c4f8c5b48" exitCode=2
Jun 15 06:04:19.904246 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.904180    2572 generic.go:356] "Generic (PLEG): container finished" podID="3b50eac5-7651-40b1-b5ef-803467f5b9de" containerID="7814e7bcc864235ce46ad30f37241d19d77971082784a0289b5a7533f7877f0f" exitCode=2
Jun 15 06:04:19.904246 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.904196    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm"
Jun 15 06:04:19.904246 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.904218    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm" event={"ID":"3b50eac5-7651-40b1-b5ef-803467f5b9de","Type":"ContainerDied","Data":"2cbd9fca427ef3bca2b01fce3fc8cc39287b86eba2039c8e9c2f919c4f8c5b48"}
Jun 15 06:04:19.904346 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.904265    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm" event={"ID":"3b50eac5-7651-40b1-b5ef-803467f5b9de","Type":"ContainerDied","Data":"7814e7bcc864235ce46ad30f37241d19d77971082784a0289b5a7533f7877f0f"}
Jun 15 06:04:19.904346 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.904288    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm" event={"ID":"3b50eac5-7651-40b1-b5ef-803467f5b9de","Type":"ContainerDied","Data":"e26448e8de804195ef3c9205cfd4de6c220cf80fbb606b176bb944400d19336c"}
Jun 15 06:04:19.904346 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.904312    2572 scope.go:117] "RemoveContainer" containerID="2cbd9fca427ef3bca2b01fce3fc8cc39287b86eba2039c8e9c2f919c4f8c5b48"
Jun 15 06:04:19.914823 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.914621    2572 scope.go:117] "RemoveContainer" containerID="7814e7bcc864235ce46ad30f37241d19d77971082784a0289b5a7533f7877f0f"
Jun 15 06:04:19.924337 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.924321    2572 scope.go:117] "RemoveContainer" containerID="2cbd9fca427ef3bca2b01fce3fc8cc39287b86eba2039c8e9c2f919c4f8c5b48"
Jun 15 06:04:19.924649 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:04:19.924629    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cbd9fca427ef3bca2b01fce3fc8cc39287b86eba2039c8e9c2f919c4f8c5b48\": container with ID starting with 2cbd9fca427ef3bca2b01fce3fc8cc39287b86eba2039c8e9c2f919c4f8c5b48 not found: ID does not exist" containerID="2cbd9fca427ef3bca2b01fce3fc8cc39287b86eba2039c8e9c2f919c4f8c5b48"
Jun 15 06:04:19.924747 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.924663    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cbd9fca427ef3bca2b01fce3fc8cc39287b86eba2039c8e9c2f919c4f8c5b48"} err="failed to get container status \"2cbd9fca427ef3bca2b01fce3fc8cc39287b86eba2039c8e9c2f919c4f8c5b48\": rpc error: code = NotFound desc = could not find container \"2cbd9fca427ef3bca2b01fce3fc8cc39287b86eba2039c8e9c2f919c4f8c5b48\": container with ID starting with 2cbd9fca427ef3bca2b01fce3fc8cc39287b86eba2039c8e9c2f919c4f8c5b48 not found: ID does not exist"
Jun 15 06:04:19.924747 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.924687    2572 scope.go:117] "RemoveContainer" containerID="7814e7bcc864235ce46ad30f37241d19d77971082784a0289b5a7533f7877f0f"
Jun 15 06:04:19.924942 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:04:19.924923    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7814e7bcc864235ce46ad30f37241d19d77971082784a0289b5a7533f7877f0f\": container with ID starting with 7814e7bcc864235ce46ad30f37241d19d77971082784a0289b5a7533f7877f0f not found: ID does not exist" containerID="7814e7bcc864235ce46ad30f37241d19d77971082784a0289b5a7533f7877f0f"
Jun 15 06:04:19.925003 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.924947    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7814e7bcc864235ce46ad30f37241d19d77971082784a0289b5a7533f7877f0f"} err="failed to get container status \"7814e7bcc864235ce46ad30f37241d19d77971082784a0289b5a7533f7877f0f\": rpc error: code = NotFound desc = could not find container \"7814e7bcc864235ce46ad30f37241d19d77971082784a0289b5a7533f7877f0f\": container with ID starting with 7814e7bcc864235ce46ad30f37241d19d77971082784a0289b5a7533f7877f0f not found: ID does not exist"
Jun 15 06:04:19.925003 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.924970    2572 scope.go:117] "RemoveContainer" containerID="2cbd9fca427ef3bca2b01fce3fc8cc39287b86eba2039c8e9c2f919c4f8c5b48"
Jun 15 06:04:19.925187 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.925169    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cbd9fca427ef3bca2b01fce3fc8cc39287b86eba2039c8e9c2f919c4f8c5b48"} err="failed to get container status \"2cbd9fca427ef3bca2b01fce3fc8cc39287b86eba2039c8e9c2f919c4f8c5b48\": rpc error: code = NotFound desc = could not find container \"2cbd9fca427ef3bca2b01fce3fc8cc39287b86eba2039c8e9c2f919c4f8c5b48\": container with ID starting with 2cbd9fca427ef3bca2b01fce3fc8cc39287b86eba2039c8e9c2f919c4f8c5b48 not found: ID does not exist"
Jun 15 06:04:19.925249 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.925198    2572 scope.go:117] "RemoveContainer" containerID="7814e7bcc864235ce46ad30f37241d19d77971082784a0289b5a7533f7877f0f"
Jun 15 06:04:19.925427 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.925410    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7814e7bcc864235ce46ad30f37241d19d77971082784a0289b5a7533f7877f0f"} err="failed to get container status \"7814e7bcc864235ce46ad30f37241d19d77971082784a0289b5a7533f7877f0f\": rpc error: code = NotFound desc = could not find container \"7814e7bcc864235ce46ad30f37241d19d77971082784a0289b5a7533f7877f0f\": container with ID starting with 7814e7bcc864235ce46ad30f37241d19d77971082784a0289b5a7533f7877f0f not found: ID does not exist"
Jun 15 06:04:19.937100 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.937068    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm"]
Jun 15 06:04:19.940936 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:19.940910    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-hbpsm"]
Jun 15 06:04:21.330074 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:21.330033    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b50eac5-7651-40b1-b5ef-803467f5b9de" path="/var/lib/kubelet/pods/3b50eac5-7651-40b1-b5ef-803467f5b9de/volumes"
Jun 15 06:04:22.478737 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:22.478693    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jun 15 06:04:23.926947 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:23.926908    2572 generic.go:356] "Generic (PLEG): container finished" podID="45cc701b-3e04-4c1e-94b8-e8ea767f96d7" containerID="eced31ebab941fa3cc681148bf4e1e12f048c283dd8bafe373548e2d34337d40" exitCode=0
Jun 15 06:04:23.927434 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:23.926981    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv" event={"ID":"45cc701b-3e04-4c1e-94b8-e8ea767f96d7","Type":"ContainerDied","Data":"eced31ebab941fa3cc681148bf4e1e12f048c283dd8bafe373548e2d34337d40"}
Jun 15 06:04:23.929758 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:23.929728    2572 generic.go:356] "Generic (PLEG): container finished" podID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerID="49b7780bb21db07b3c7f40013bae09841cc4c305d6a9b154c36147d891686c12" exitCode=0
Jun 15 06:04:23.929853 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:23.929766    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" event={"ID":"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1","Type":"ContainerDied","Data":"49b7780bb21db07b3c7f40013bae09841cc4c305d6a9b154c36147d891686c12"}
Jun 15 06:04:27.478787 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:27.478735    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jun 15 06:04:27.482712 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:27.482672    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:04:27.483974 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:27.483944    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jun 15 06:04:30.968265 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:30.968229    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv" event={"ID":"45cc701b-3e04-4c1e-94b8-e8ea767f96d7","Type":"ContainerStarted","Data":"80e188060876ba51ce8b4afc2d272859800b8d83de8e270ce8d2e0a8d8a63c6a"}
Jun 15 06:04:30.968628 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:30.968290    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv" event={"ID":"45cc701b-3e04-4c1e-94b8-e8ea767f96d7","Type":"ContainerStarted","Data":"eaa5ed8df38fb0c8fd23b088d5fe9aacf2aa1c80355523913d278348842a48c4"}
Jun 15 06:04:30.968628 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:30.968552    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv"
Jun 15 06:04:30.989925 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:30.989877    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv" podStartSLOduration=2.4111939700000002 podStartE2EDuration="12.989859422s" totalImagesPullingTime="6.560761451s" totalInitContainerRuntime="4.017904001s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:04:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:04:30.988631443 +0000 UTC m=+986.136811439" watchObservedRunningTime="2026-06-15 06:04:30.989859422 +0000 UTC m=+986.138039419"
Jun 15 06:04:31.973863 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:31.973825    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv"
Jun 15 06:04:31.975093 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:31.975068    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv" podUID="45cc701b-3e04-4c1e-94b8-e8ea767f96d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jun 15 06:04:32.478472 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:32.478428    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jun 15 06:04:32.478666 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:32.478578    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"
Jun 15 06:04:32.978657 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:32.978624    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv" podUID="45cc701b-3e04-4c1e-94b8-e8ea767f96d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jun 15 06:04:37.478052 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:37.478001    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jun 15 06:04:37.482750 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:37.482719    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:04:37.483895 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:37.483867    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jun 15 06:04:37.984441 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:37.984414    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv"
Jun 15 06:04:37.984939 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:37.984915    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv" podUID="45cc701b-3e04-4c1e-94b8-e8ea767f96d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jun 15 06:04:42.478703 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:42.478659    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jun 15 06:04:47.478409 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:47.478352    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.32:8643/healthz\": dial tcp 10.134.0.32:8643: connect: connection refused"
Jun 15 06:04:47.483331 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:47.483290    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jun 15 06:04:47.483482 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:47.483462    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"
Jun 15 06:04:47.483550 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:47.483513    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jun 15 06:04:47.483709 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:47.483694    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"
Jun 15 06:04:47.984822 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:47.984788    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv" podUID="45cc701b-3e04-4c1e-94b8-e8ea767f96d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jun 15 06:04:48.920703 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:48.920680    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"
Jun 15 06:04:49.056223 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.056134    2572 generic.go:356] "Generic (PLEG): container finished" podID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerID="b45759f69f7b3372b3dfe909adc59e538ed52b526f41377fe7babecc45579dec" exitCode=0
Jun 15 06:04:49.056223 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.056196    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" event={"ID":"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1","Type":"ContainerDied","Data":"b45759f69f7b3372b3dfe909adc59e538ed52b526f41377fe7babecc45579dec"}
Jun 15 06:04:49.056411 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.056199    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"
Jun 15 06:04:49.056411 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.056281    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp" event={"ID":"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1","Type":"ContainerDied","Data":"5c69ea630217ca37df49651d187345780243bafd10cb072feca08cd324c66fbe"}
Jun 15 06:04:49.056411 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.056313    2572 scope.go:117] "RemoveContainer" containerID="b45759f69f7b3372b3dfe909adc59e538ed52b526f41377fe7babecc45579dec"
Jun 15 06:04:49.065273 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.065250    2572 scope.go:117] "RemoveContainer" containerID="fe10bb79c344545fdf56a4ee3467eecec9b7fceba89e0335300876067a547e45"
Jun 15 06:04:49.072465 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.072447    2572 scope.go:117] "RemoveContainer" containerID="49b7780bb21db07b3c7f40013bae09841cc4c305d6a9b154c36147d891686c12"
Jun 15 06:04:49.079578 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.079559    2572 scope.go:117] "RemoveContainer" containerID="2c815aac7a17570637fec2292b578bda64471fa09e92cb193a1bc7899d1046af"
Jun 15 06:04:49.086666 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.086651    2572 scope.go:117] "RemoveContainer" containerID="b45759f69f7b3372b3dfe909adc59e538ed52b526f41377fe7babecc45579dec"
Jun 15 06:04:49.086910 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:04:49.086884    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b45759f69f7b3372b3dfe909adc59e538ed52b526f41377fe7babecc45579dec\": container with ID starting with b45759f69f7b3372b3dfe909adc59e538ed52b526f41377fe7babecc45579dec not found: ID does not exist" containerID="b45759f69f7b3372b3dfe909adc59e538ed52b526f41377fe7babecc45579dec"
Jun 15 06:04:49.086994 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.086921    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b45759f69f7b3372b3dfe909adc59e538ed52b526f41377fe7babecc45579dec"} err="failed to get container status \"b45759f69f7b3372b3dfe909adc59e538ed52b526f41377fe7babecc45579dec\": rpc error: code = NotFound desc = could not find container \"b45759f69f7b3372b3dfe909adc59e538ed52b526f41377fe7babecc45579dec\": container with ID starting with b45759f69f7b3372b3dfe909adc59e538ed52b526f41377fe7babecc45579dec not found: ID does not exist"
Jun 15 06:04:49.086994 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.086945    2572 scope.go:117] "RemoveContainer" containerID="fe10bb79c344545fdf56a4ee3467eecec9b7fceba89e0335300876067a547e45"
Jun 15 06:04:49.086994 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.086984    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-kserve-provision-location\") pod \"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1\" (UID: \"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1\") "
Jun 15 06:04:49.087203 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.087017    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xl6tz\" (UniqueName: \"kubernetes.io/projected/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-kube-api-access-xl6tz\") pod \"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1\" (UID: \"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1\") "
Jun 15 06:04:49.087203 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.087088    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-kube-rbac-proxy-sar-config\") pod \"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1\" (UID: \"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1\") "
Jun 15 06:04:49.087203 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.087188    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-cabundle-cert\") pod \"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1\" (UID: \"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1\") "
Jun 15 06:04:49.087405 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.087223    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-proxy-tls\") pod \"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1\" (UID: \"211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1\") "
Jun 15 06:04:49.087405 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:04:49.087245    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe10bb79c344545fdf56a4ee3467eecec9b7fceba89e0335300876067a547e45\": container with ID starting with fe10bb79c344545fdf56a4ee3467eecec9b7fceba89e0335300876067a547e45 not found: ID does not exist" containerID="fe10bb79c344545fdf56a4ee3467eecec9b7fceba89e0335300876067a547e45"
Jun 15 06:04:49.087405 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.087276    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe10bb79c344545fdf56a4ee3467eecec9b7fceba89e0335300876067a547e45"} err="failed to get container status \"fe10bb79c344545fdf56a4ee3467eecec9b7fceba89e0335300876067a547e45\": rpc error: code = NotFound desc = could not find container \"fe10bb79c344545fdf56a4ee3467eecec9b7fceba89e0335300876067a547e45\": container with ID starting with fe10bb79c344545fdf56a4ee3467eecec9b7fceba89e0335300876067a547e45 not found: ID does not exist"
Jun 15 06:04:49.087405 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.087296    2572 scope.go:117] "RemoveContainer" containerID="49b7780bb21db07b3c7f40013bae09841cc4c305d6a9b154c36147d891686c12"
Jun 15 06:04:49.087405 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.087336    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" (UID: "211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:04:49.087643 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.087424    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" (UID: "211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:04:49.087643 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.087577    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:04:49.087643 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:04:49.087580    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49b7780bb21db07b3c7f40013bae09841cc4c305d6a9b154c36147d891686c12\": container with ID starting with 49b7780bb21db07b3c7f40013bae09841cc4c305d6a9b154c36147d891686c12 not found: ID does not exist" containerID="49b7780bb21db07b3c7f40013bae09841cc4c305d6a9b154c36147d891686c12"
Jun 15 06:04:49.087643 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.087597    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:04:49.087643 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.087609    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49b7780bb21db07b3c7f40013bae09841cc4c305d6a9b154c36147d891686c12"} err="failed to get container status \"49b7780bb21db07b3c7f40013bae09841cc4c305d6a9b154c36147d891686c12\": rpc error: code = NotFound desc = could not find container \"49b7780bb21db07b3c7f40013bae09841cc4c305d6a9b154c36147d891686c12\": container with ID starting with 49b7780bb21db07b3c7f40013bae09841cc4c305d6a9b154c36147d891686c12 not found: ID does not exist"
Jun 15 06:04:49.087643 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.087629    2572 scope.go:117] "RemoveContainer" containerID="2c815aac7a17570637fec2292b578bda64471fa09e92cb193a1bc7899d1046af"
Jun 15 06:04:49.087892 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.087691    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" (UID: "211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:04:49.087950 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:04:49.087885    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c815aac7a17570637fec2292b578bda64471fa09e92cb193a1bc7899d1046af\": container with ID starting with 2c815aac7a17570637fec2292b578bda64471fa09e92cb193a1bc7899d1046af not found: ID does not exist" containerID="2c815aac7a17570637fec2292b578bda64471fa09e92cb193a1bc7899d1046af"
Jun 15 06:04:49.087950 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.087916    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c815aac7a17570637fec2292b578bda64471fa09e92cb193a1bc7899d1046af"} err="failed to get container status \"2c815aac7a17570637fec2292b578bda64471fa09e92cb193a1bc7899d1046af\": rpc error: code = NotFound desc = could not find container \"2c815aac7a17570637fec2292b578bda64471fa09e92cb193a1bc7899d1046af\": container with ID starting with 2c815aac7a17570637fec2292b578bda64471fa09e92cb193a1bc7899d1046af not found: ID does not exist"
Jun 15 06:04:49.089028 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.089007    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-kube-api-access-xl6tz" (OuterVolumeSpecName: "kube-api-access-xl6tz") pod "211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" (UID: "211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1"). InnerVolumeSpecName "kube-api-access-xl6tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:04:49.089292 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.089268    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" (UID: "211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:04:49.188369 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.188331    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:04:49.188532 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.188386    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:04:49.188532 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.188395    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl6tz\" (UniqueName: \"kubernetes.io/projected/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1-kube-api-access-xl6tz\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:04:49.375351 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.375320    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"]
Jun 15 06:04:49.380315 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:49.380288    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-654b69f66-snppp"]
Jun 15 06:04:51.323894 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:51.323861    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" path="/var/lib/kubelet/pods/211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1/volumes"
Jun 15 06:04:57.985347 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:04:57.985308    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv" podUID="45cc701b-3e04-4c1e-94b8-e8ea767f96d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jun 15 06:05:07.985352 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:07.985313    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv" podUID="45cc701b-3e04-4c1e-94b8-e8ea767f96d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jun 15 06:05:17.985190 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:17.985136    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv" podUID="45cc701b-3e04-4c1e-94b8-e8ea767f96d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jun 15 06:05:27.985495 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:27.985457    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv" podUID="45cc701b-3e04-4c1e-94b8-e8ea767f96d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jun 15 06:05:37.985095 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:37.985056    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv" podUID="45cc701b-3e04-4c1e-94b8-e8ea767f96d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jun 15 06:05:47.986222 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:47.986148    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv"
Jun 15 06:05:48.962628 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:48.962599    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv"]
Jun 15 06:05:48.962840 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:48.962815    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv" podUID="45cc701b-3e04-4c1e-94b8-e8ea767f96d7" containerName="kserve-container" containerID="cri-o://eaa5ed8df38fb0c8fd23b088d5fe9aacf2aa1c80355523913d278348842a48c4" gracePeriod=30
Jun 15 06:05:48.962927 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:48.962860    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv" podUID="45cc701b-3e04-4c1e-94b8-e8ea767f96d7" containerName="kube-rbac-proxy" containerID="cri-o://80e188060876ba51ce8b4afc2d272859800b8d83de8e270ce8d2e0a8d8a63c6a" gracePeriod=30
Jun 15 06:05:49.075882 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.075850    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"]
Jun 15 06:05:49.076229 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.076204    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b50eac5-7651-40b1-b5ef-803467f5b9de" containerName="kserve-container"
Jun 15 06:05:49.076229 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.076216    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b50eac5-7651-40b1-b5ef-803467f5b9de" containerName="kserve-container"
Jun 15 06:05:49.076229 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.076226    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="storage-initializer"
Jun 15 06:05:49.076331 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.076233    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="storage-initializer"
Jun 15 06:05:49.076331 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.076240    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="kube-rbac-proxy"
Jun 15 06:05:49.076331 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.076246    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="kube-rbac-proxy"
Jun 15 06:05:49.076331 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.076257    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="kserve-container"
Jun 15 06:05:49.076331 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.076262    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="kserve-container"
Jun 15 06:05:49.076331 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.076270    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b50eac5-7651-40b1-b5ef-803467f5b9de" containerName="kube-rbac-proxy"
Jun 15 06:05:49.076331 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.076275    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b50eac5-7651-40b1-b5ef-803467f5b9de" containerName="kube-rbac-proxy"
Jun 15 06:05:49.076331 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.076284    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="agent"
Jun 15 06:05:49.076331 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.076289    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="agent"
Jun 15 06:05:49.076624 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.076337    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b50eac5-7651-40b1-b5ef-803467f5b9de" containerName="kserve-container"
Jun 15 06:05:49.076624 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.076346    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="kserve-container"
Jun 15 06:05:49.076624 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.076380    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="agent"
Jun 15 06:05:49.076624 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.076388    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="211f83c0-34ef-4cb3-8ae7-cf2c6a9889e1" containerName="kube-rbac-proxy"
Jun 15 06:05:49.076624 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.076396    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b50eac5-7651-40b1-b5ef-803467f5b9de" containerName="kube-rbac-proxy"
Jun 15 06:05:49.078795 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.078777    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"
Jun 15 06:05:49.081032 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.081013    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Jun 15 06:05:49.081141 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.081045    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Jun 15 06:05:49.091468 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.091444    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"]
Jun 15 06:05:49.131119 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.131087    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/893a986a-28a8-4f2e-9998-ec7cd7182c10-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt\" (UID: \"893a986a-28a8-4f2e-9998-ec7cd7182c10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"
Jun 15 06:05:49.131119 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.131120    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/893a986a-28a8-4f2e-9998-ec7cd7182c10-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt\" (UID: \"893a986a-28a8-4f2e-9998-ec7cd7182c10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"
Jun 15 06:05:49.131354 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.131227    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/893a986a-28a8-4f2e-9998-ec7cd7182c10-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt\" (UID: \"893a986a-28a8-4f2e-9998-ec7cd7182c10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"
Jun 15 06:05:49.131354 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.131293    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp77h\" (UniqueName: \"kubernetes.io/projected/893a986a-28a8-4f2e-9998-ec7cd7182c10-kube-api-access-hp77h\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt\" (UID: \"893a986a-28a8-4f2e-9998-ec7cd7182c10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"
Jun 15 06:05:49.131354 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.131338    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/893a986a-28a8-4f2e-9998-ec7cd7182c10-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt\" (UID: \"893a986a-28a8-4f2e-9998-ec7cd7182c10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"
Jun 15 06:05:49.232739 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.232655    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp77h\" (UniqueName: \"kubernetes.io/projected/893a986a-28a8-4f2e-9998-ec7cd7182c10-kube-api-access-hp77h\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt\" (UID: \"893a986a-28a8-4f2e-9998-ec7cd7182c10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"
Jun 15 06:05:49.232739 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.232703    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/893a986a-28a8-4f2e-9998-ec7cd7182c10-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt\" (UID: \"893a986a-28a8-4f2e-9998-ec7cd7182c10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"
Jun 15 06:05:49.232739 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.232723    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/893a986a-28a8-4f2e-9998-ec7cd7182c10-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt\" (UID: \"893a986a-28a8-4f2e-9998-ec7cd7182c10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"
Jun 15 06:05:49.232739 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.232738    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/893a986a-28a8-4f2e-9998-ec7cd7182c10-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt\" (UID: \"893a986a-28a8-4f2e-9998-ec7cd7182c10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"
Jun 15 06:05:49.233041 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.232806    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/893a986a-28a8-4f2e-9998-ec7cd7182c10-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt\" (UID: \"893a986a-28a8-4f2e-9998-ec7cd7182c10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"
Jun 15 06:05:49.233041 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:05:49.232823    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-serving-cert: secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Jun 15 06:05:49.233041 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:05:49.232903    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/893a986a-28a8-4f2e-9998-ec7cd7182c10-proxy-tls podName:893a986a-28a8-4f2e-9998-ec7cd7182c10 nodeName:}" failed. No retries permitted until 2026-06-15 06:05:49.732883951 +0000 UTC m=+1064.881063930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/893a986a-28a8-4f2e-9998-ec7cd7182c10-proxy-tls") pod "isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt" (UID: "893a986a-28a8-4f2e-9998-ec7cd7182c10") : secret "isvc-lightgbm-runtime-predictor-serving-cert" not found
Jun 15 06:05:49.233210 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.233196    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/893a986a-28a8-4f2e-9998-ec7cd7182c10-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt\" (UID: \"893a986a-28a8-4f2e-9998-ec7cd7182c10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"
Jun 15 06:05:49.233472 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.233449    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/893a986a-28a8-4f2e-9998-ec7cd7182c10-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt\" (UID: \"893a986a-28a8-4f2e-9998-ec7cd7182c10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"
Jun 15 06:05:49.233575 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.233529    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/893a986a-28a8-4f2e-9998-ec7cd7182c10-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt\" (UID: \"893a986a-28a8-4f2e-9998-ec7cd7182c10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"
Jun 15 06:05:49.242121 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.242095    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp77h\" (UniqueName: \"kubernetes.io/projected/893a986a-28a8-4f2e-9998-ec7cd7182c10-kube-api-access-hp77h\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt\" (UID: \"893a986a-28a8-4f2e-9998-ec7cd7182c10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"
Jun 15 06:05:49.349352 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.349315    2572 generic.go:356] "Generic (PLEG): container finished" podID="45cc701b-3e04-4c1e-94b8-e8ea767f96d7" containerID="80e188060876ba51ce8b4afc2d272859800b8d83de8e270ce8d2e0a8d8a63c6a" exitCode=2
Jun 15 06:05:49.349528 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.349390    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv" event={"ID":"45cc701b-3e04-4c1e-94b8-e8ea767f96d7","Type":"ContainerDied","Data":"80e188060876ba51ce8b4afc2d272859800b8d83de8e270ce8d2e0a8d8a63c6a"}
Jun 15 06:05:49.737928 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.737892    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/893a986a-28a8-4f2e-9998-ec7cd7182c10-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt\" (UID: \"893a986a-28a8-4f2e-9998-ec7cd7182c10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"
Jun 15 06:05:49.740496 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.740468    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/893a986a-28a8-4f2e-9998-ec7cd7182c10-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt\" (UID: \"893a986a-28a8-4f2e-9998-ec7cd7182c10\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"
Jun 15 06:05:49.989134 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:49.989038    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"
Jun 15 06:05:50.127161 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:50.127086    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"]
Jun 15 06:05:50.355517 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:50.355439    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt" event={"ID":"893a986a-28a8-4f2e-9998-ec7cd7182c10","Type":"ContainerStarted","Data":"c080cc4bd45788fc415ea3bf505a1c6e1202978958632c87486422fe36b45ec8"}
Jun 15 06:05:50.355517 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:50.355495    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt" event={"ID":"893a986a-28a8-4f2e-9998-ec7cd7182c10","Type":"ContainerStarted","Data":"75c55ed374dd91398e592c7b7ce88898ca5003ff0c85ef1a0c6828a12873b68c"}
Jun 15 06:05:52.979580 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:52.979532    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv" podUID="45cc701b-3e04-4c1e-94b8-e8ea767f96d7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jun 15 06:05:53.370866 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:53.370838    2572 generic.go:356] "Generic (PLEG): container finished" podID="45cc701b-3e04-4c1e-94b8-e8ea767f96d7" containerID="eaa5ed8df38fb0c8fd23b088d5fe9aacf2aa1c80355523913d278348842a48c4" exitCode=0
Jun 15 06:05:53.371011 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:53.370873    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv" event={"ID":"45cc701b-3e04-4c1e-94b8-e8ea767f96d7","Type":"ContainerDied","Data":"eaa5ed8df38fb0c8fd23b088d5fe9aacf2aa1c80355523913d278348842a48c4"}
Jun 15 06:05:53.400082 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:53.400062    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv"
Jun 15 06:05:53.476113 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:53.476079    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-kserve-provision-location\") pod \"45cc701b-3e04-4c1e-94b8-e8ea767f96d7\" (UID: \"45cc701b-3e04-4c1e-94b8-e8ea767f96d7\") "
Jun 15 06:05:53.476264 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:53.476128    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-cabundle-cert\") pod \"45cc701b-3e04-4c1e-94b8-e8ea767f96d7\" (UID: \"45cc701b-3e04-4c1e-94b8-e8ea767f96d7\") "
Jun 15 06:05:53.476264 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:53.476156    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njbwg\" (UniqueName: \"kubernetes.io/projected/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-kube-api-access-njbwg\") pod \"45cc701b-3e04-4c1e-94b8-e8ea767f96d7\" (UID: \"45cc701b-3e04-4c1e-94b8-e8ea767f96d7\") "
Jun 15 06:05:53.476264 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:53.476174    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-kube-rbac-proxy-sar-config\") pod \"45cc701b-3e04-4c1e-94b8-e8ea767f96d7\" (UID: \"45cc701b-3e04-4c1e-94b8-e8ea767f96d7\") "
Jun 15 06:05:53.476264 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:53.476199    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-proxy-tls\") pod \"45cc701b-3e04-4c1e-94b8-e8ea767f96d7\" (UID: \"45cc701b-3e04-4c1e-94b8-e8ea767f96d7\") "
Jun 15 06:05:53.476536 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:53.476453    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "45cc701b-3e04-4c1e-94b8-e8ea767f96d7" (UID: "45cc701b-3e04-4c1e-94b8-e8ea767f96d7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:05:53.476597 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:53.476563    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "45cc701b-3e04-4c1e-94b8-e8ea767f96d7" (UID: "45cc701b-3e04-4c1e-94b8-e8ea767f96d7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:05:53.476640 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:53.476588    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "45cc701b-3e04-4c1e-94b8-e8ea767f96d7" (UID: "45cc701b-3e04-4c1e-94b8-e8ea767f96d7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:05:53.478321 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:53.478300    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-kube-api-access-njbwg" (OuterVolumeSpecName: "kube-api-access-njbwg") pod "45cc701b-3e04-4c1e-94b8-e8ea767f96d7" (UID: "45cc701b-3e04-4c1e-94b8-e8ea767f96d7"). InnerVolumeSpecName "kube-api-access-njbwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:05:53.478417 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:53.478346    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "45cc701b-3e04-4c1e-94b8-e8ea767f96d7" (UID: "45cc701b-3e04-4c1e-94b8-e8ea767f96d7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:05:53.577190 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:53.577111    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:05:53.577190 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:53.577142    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njbwg\" (UniqueName: \"kubernetes.io/projected/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-kube-api-access-njbwg\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:05:53.577190 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:53.577153    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:05:53.577190 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:53.577162    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:05:53.577190 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:53.577173    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/45cc701b-3e04-4c1e-94b8-e8ea767f96d7-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:05:54.377343 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:54.377267    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv"
Jun 15 06:05:54.377343 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:54.377289    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv" event={"ID":"45cc701b-3e04-4c1e-94b8-e8ea767f96d7","Type":"ContainerDied","Data":"81fda7e551bf458a87c6d865187339194042b652ed7b2d58e344e52eb30fb452"}
Jun 15 06:05:54.377824 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:54.377379    2572 scope.go:117] "RemoveContainer" containerID="80e188060876ba51ce8b4afc2d272859800b8d83de8e270ce8d2e0a8d8a63c6a"
Jun 15 06:05:54.378938 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:54.378919    2572 generic.go:356] "Generic (PLEG): container finished" podID="893a986a-28a8-4f2e-9998-ec7cd7182c10" containerID="c080cc4bd45788fc415ea3bf505a1c6e1202978958632c87486422fe36b45ec8" exitCode=0
Jun 15 06:05:54.379028 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:54.378993    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt" event={"ID":"893a986a-28a8-4f2e-9998-ec7cd7182c10","Type":"ContainerDied","Data":"c080cc4bd45788fc415ea3bf505a1c6e1202978958632c87486422fe36b45ec8"}
Jun 15 06:05:54.386424 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:54.386034    2572 scope.go:117] "RemoveContainer" containerID="eaa5ed8df38fb0c8fd23b088d5fe9aacf2aa1c80355523913d278348842a48c4"
Jun 15 06:05:54.394569 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:54.394543    2572 scope.go:117] "RemoveContainer" containerID="eced31ebab941fa3cc681148bf4e1e12f048c283dd8bafe373548e2d34337d40"
Jun 15 06:05:54.415601 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:54.415571    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv"]
Jun 15 06:05:54.419946 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:54.419924    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-nr8lv"]
Jun 15 06:05:55.323852 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:55.323813    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45cc701b-3e04-4c1e-94b8-e8ea767f96d7" path="/var/lib/kubelet/pods/45cc701b-3e04-4c1e-94b8-e8ea767f96d7/volumes"
Jun 15 06:05:55.386660 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:55.386629    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt" event={"ID":"893a986a-28a8-4f2e-9998-ec7cd7182c10","Type":"ContainerStarted","Data":"b952be65a451562bdd989584438ab284d0e805cd5ff68938b0dd1ffd1c89892e"}
Jun 15 06:05:55.387028 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:55.386691    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt" event={"ID":"893a986a-28a8-4f2e-9998-ec7cd7182c10","Type":"ContainerStarted","Data":"5d1bf137c266f4d9b7f27c2550a8e445c75d57ec64ad343a116935665163255e"}
Jun 15 06:05:55.387028 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:55.386965    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"
Jun 15 06:05:55.387117 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:55.387088    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"
Jun 15 06:05:55.388421 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:55.388394    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt" podUID="893a986a-28a8-4f2e-9998-ec7cd7182c10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jun 15 06:05:55.410550 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:55.410500    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt" podStartSLOduration=2.499533243 podStartE2EDuration="6.410485278s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.910952035s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:05:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:05:55.408906335 +0000 UTC m=+1070.557086343" watchObservedRunningTime="2026-06-15 06:05:55.410485278 +0000 UTC m=+1070.558665275"
Jun 15 06:05:56.391027 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:05:56.390985    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt" podUID="893a986a-28a8-4f2e-9998-ec7cd7182c10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jun 15 06:06:01.395208 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:06:01.395176    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"
Jun 15 06:06:01.395769 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:06:01.395642    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt" podUID="893a986a-28a8-4f2e-9998-ec7cd7182c10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jun 15 06:06:11.395829 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:06:11.395793    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt" podUID="893a986a-28a8-4f2e-9998-ec7cd7182c10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jun 15 06:06:21.395662 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:06:21.395622    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt" podUID="893a986a-28a8-4f2e-9998-ec7cd7182c10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jun 15 06:06:31.396222 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:06:31.396181    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt" podUID="893a986a-28a8-4f2e-9998-ec7cd7182c10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jun 15 06:06:41.396597 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:06:41.396556    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt" podUID="893a986a-28a8-4f2e-9998-ec7cd7182c10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jun 15 06:06:51.396060 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:06:51.396014    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt" podUID="893a986a-28a8-4f2e-9998-ec7cd7182c10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jun 15 06:07:01.395928 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:01.395890    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt" podUID="893a986a-28a8-4f2e-9998-ec7cd7182c10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jun 15 06:07:11.396537 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:11.396507    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"
Jun 15 06:07:19.819144 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:19.819072    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"]
Jun 15 06:07:19.819661 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:19.819327    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt" podUID="893a986a-28a8-4f2e-9998-ec7cd7182c10" containerName="kserve-container" containerID="cri-o://5d1bf137c266f4d9b7f27c2550a8e445c75d57ec64ad343a116935665163255e" gracePeriod=30
Jun 15 06:07:19.819661 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:19.819413    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt" podUID="893a986a-28a8-4f2e-9998-ec7cd7182c10" containerName="kube-rbac-proxy" containerID="cri-o://b952be65a451562bdd989584438ab284d0e805cd5ff68938b0dd1ffd1c89892e" gracePeriod=30
Jun 15 06:07:19.901239 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:19.901205    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"]
Jun 15 06:07:19.901595 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:19.901578    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45cc701b-3e04-4c1e-94b8-e8ea767f96d7" containerName="storage-initializer"
Jun 15 06:07:19.901595 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:19.901593    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="45cc701b-3e04-4c1e-94b8-e8ea767f96d7" containerName="storage-initializer"
Jun 15 06:07:19.901736 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:19.901606    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45cc701b-3e04-4c1e-94b8-e8ea767f96d7" containerName="kserve-container"
Jun 15 06:07:19.901736 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:19.901612    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="45cc701b-3e04-4c1e-94b8-e8ea767f96d7" containerName="kserve-container"
Jun 15 06:07:19.901736 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:19.901626    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45cc701b-3e04-4c1e-94b8-e8ea767f96d7" containerName="kube-rbac-proxy"
Jun 15 06:07:19.901736 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:19.901632    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="45cc701b-3e04-4c1e-94b8-e8ea767f96d7" containerName="kube-rbac-proxy"
Jun 15 06:07:19.901736 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:19.901680    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="45cc701b-3e04-4c1e-94b8-e8ea767f96d7" containerName="kserve-container"
Jun 15 06:07:19.901736 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:19.901691    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="45cc701b-3e04-4c1e-94b8-e8ea767f96d7" containerName="kube-rbac-proxy"
Jun 15 06:07:19.904010 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:19.903994    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"
Jun 15 06:07:19.906158 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:19.906139    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-kube-rbac-proxy-sar-config\""
Jun 15 06:07:19.906252 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:19.906163    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-runtime-predictor-serving-cert\""
Jun 15 06:07:19.913912 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:19.913887    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"]
Jun 15 06:07:19.955887 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:19.955848    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3240d84a-03d9-45cc-98f3-66ef7e6f8381-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g\" (UID: \"3240d84a-03d9-45cc-98f3-66ef7e6f8381\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"
Jun 15 06:07:19.955887 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:19.955887    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3240d84a-03d9-45cc-98f3-66ef7e6f8381-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g\" (UID: \"3240d84a-03d9-45cc-98f3-66ef7e6f8381\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"
Jun 15 06:07:19.956070 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:19.955918    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3240d84a-03d9-45cc-98f3-66ef7e6f8381-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g\" (UID: \"3240d84a-03d9-45cc-98f3-66ef7e6f8381\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"
Jun 15 06:07:19.956070 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:19.956020    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnnl8\" (UniqueName: \"kubernetes.io/projected/3240d84a-03d9-45cc-98f3-66ef7e6f8381-kube-api-access-gnnl8\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g\" (UID: \"3240d84a-03d9-45cc-98f3-66ef7e6f8381\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"
Jun 15 06:07:19.956070 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:19.956055    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3240d84a-03d9-45cc-98f3-66ef7e6f8381-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g\" (UID: \"3240d84a-03d9-45cc-98f3-66ef7e6f8381\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"
Jun 15 06:07:20.056921 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:20.056880    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3240d84a-03d9-45cc-98f3-66ef7e6f8381-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g\" (UID: \"3240d84a-03d9-45cc-98f3-66ef7e6f8381\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"
Jun 15 06:07:20.056921 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:20.056922    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3240d84a-03d9-45cc-98f3-66ef7e6f8381-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g\" (UID: \"3240d84a-03d9-45cc-98f3-66ef7e6f8381\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"
Jun 15 06:07:20.057181 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:20.056947    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3240d84a-03d9-45cc-98f3-66ef7e6f8381-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g\" (UID: \"3240d84a-03d9-45cc-98f3-66ef7e6f8381\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"
Jun 15 06:07:20.057181 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:20.057009    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnnl8\" (UniqueName: \"kubernetes.io/projected/3240d84a-03d9-45cc-98f3-66ef7e6f8381-kube-api-access-gnnl8\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g\" (UID: \"3240d84a-03d9-45cc-98f3-66ef7e6f8381\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"
Jun 15 06:07:20.057181 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:20.057037    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3240d84a-03d9-45cc-98f3-66ef7e6f8381-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g\" (UID: \"3240d84a-03d9-45cc-98f3-66ef7e6f8381\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"
Jun 15 06:07:20.057181 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:07:20.057055    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-serving-cert: secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Jun 15 06:07:20.057181 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:07:20.057121    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3240d84a-03d9-45cc-98f3-66ef7e6f8381-proxy-tls podName:3240d84a-03d9-45cc-98f3-66ef7e6f8381 nodeName:}" failed. No retries permitted until 2026-06-15 06:07:20.557100414 +0000 UTC m=+1155.705280405 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3240d84a-03d9-45cc-98f3-66ef7e6f8381-proxy-tls") pod "isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g" (UID: "3240d84a-03d9-45cc-98f3-66ef7e6f8381") : secret "isvc-lightgbm-v2-runtime-predictor-serving-cert" not found
Jun 15 06:07:20.057563 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:20.057535    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3240d84a-03d9-45cc-98f3-66ef7e6f8381-kserve-provision-location\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g\" (UID: \"3240d84a-03d9-45cc-98f3-66ef7e6f8381\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"
Jun 15 06:07:20.057776 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:20.057758    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3240d84a-03d9-45cc-98f3-66ef7e6f8381-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g\" (UID: \"3240d84a-03d9-45cc-98f3-66ef7e6f8381\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"
Jun 15 06:07:20.057851 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:20.057781    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3240d84a-03d9-45cc-98f3-66ef7e6f8381-cabundle-cert\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g\" (UID: \"3240d84a-03d9-45cc-98f3-66ef7e6f8381\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"
Jun 15 06:07:20.065554 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:20.065529    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnnl8\" (UniqueName: \"kubernetes.io/projected/3240d84a-03d9-45cc-98f3-66ef7e6f8381-kube-api-access-gnnl8\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g\" (UID: \"3240d84a-03d9-45cc-98f3-66ef7e6f8381\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"
Jun 15 06:07:20.562462 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:20.562420    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3240d84a-03d9-45cc-98f3-66ef7e6f8381-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g\" (UID: \"3240d84a-03d9-45cc-98f3-66ef7e6f8381\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"
Jun 15 06:07:20.564787 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:20.564759    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3240d84a-03d9-45cc-98f3-66ef7e6f8381-proxy-tls\") pod \"isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g\" (UID: \"3240d84a-03d9-45cc-98f3-66ef7e6f8381\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"
Jun 15 06:07:20.783795 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:20.783764    2572 generic.go:356] "Generic (PLEG): container finished" podID="893a986a-28a8-4f2e-9998-ec7cd7182c10" containerID="b952be65a451562bdd989584438ab284d0e805cd5ff68938b0dd1ffd1c89892e" exitCode=2
Jun 15 06:07:20.783956 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:20.783840    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt" event={"ID":"893a986a-28a8-4f2e-9998-ec7cd7182c10","Type":"ContainerDied","Data":"b952be65a451562bdd989584438ab284d0e805cd5ff68938b0dd1ffd1c89892e"}
Jun 15 06:07:20.814677 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:20.814601    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"
Jun 15 06:07:20.946870 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:20.946848    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"]
Jun 15 06:07:21.391633 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:21.391595    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt" podUID="893a986a-28a8-4f2e-9998-ec7cd7182c10" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jun 15 06:07:21.395874 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:21.395848    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt" podUID="893a986a-28a8-4f2e-9998-ec7cd7182c10" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jun 15 06:07:21.790073 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:21.789973    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g" event={"ID":"3240d84a-03d9-45cc-98f3-66ef7e6f8381","Type":"ContainerStarted","Data":"e5c539963a2abcc81cf296bff7fd8168cfc3f441305acf62c7e348928b826e51"}
Jun 15 06:07:21.790073 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:21.790037    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g" event={"ID":"3240d84a-03d9-45cc-98f3-66ef7e6f8381","Type":"ContainerStarted","Data":"331c428f88c0c77e8ec3e6c5dbdb5c5cf53985d60ee2a41817e3e2fe77fdf21e"}
Jun 15 06:07:24.463801 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.463773    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"
Jun 15 06:07:24.500852 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.500776    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/893a986a-28a8-4f2e-9998-ec7cd7182c10-cabundle-cert\") pod \"893a986a-28a8-4f2e-9998-ec7cd7182c10\" (UID: \"893a986a-28a8-4f2e-9998-ec7cd7182c10\") "
Jun 15 06:07:24.500978 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.500863    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/893a986a-28a8-4f2e-9998-ec7cd7182c10-proxy-tls\") pod \"893a986a-28a8-4f2e-9998-ec7cd7182c10\" (UID: \"893a986a-28a8-4f2e-9998-ec7cd7182c10\") "
Jun 15 06:07:24.500978 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.500890    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hp77h\" (UniqueName: \"kubernetes.io/projected/893a986a-28a8-4f2e-9998-ec7cd7182c10-kube-api-access-hp77h\") pod \"893a986a-28a8-4f2e-9998-ec7cd7182c10\" (UID: \"893a986a-28a8-4f2e-9998-ec7cd7182c10\") "
Jun 15 06:07:24.500978 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.500953    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/893a986a-28a8-4f2e-9998-ec7cd7182c10-kube-rbac-proxy-sar-config\") pod \"893a986a-28a8-4f2e-9998-ec7cd7182c10\" (UID: \"893a986a-28a8-4f2e-9998-ec7cd7182c10\") "
Jun 15 06:07:24.501148 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.501003    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/893a986a-28a8-4f2e-9998-ec7cd7182c10-kserve-provision-location\") pod \"893a986a-28a8-4f2e-9998-ec7cd7182c10\" (UID: \"893a986a-28a8-4f2e-9998-ec7cd7182c10\") "
Jun 15 06:07:24.501148 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.501089    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/893a986a-28a8-4f2e-9998-ec7cd7182c10-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "893a986a-28a8-4f2e-9998-ec7cd7182c10" (UID: "893a986a-28a8-4f2e-9998-ec7cd7182c10"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:07:24.501321 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.501298    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/893a986a-28a8-4f2e-9998-ec7cd7182c10-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "893a986a-28a8-4f2e-9998-ec7cd7182c10" (UID: "893a986a-28a8-4f2e-9998-ec7cd7182c10"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:07:24.501405 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.501352    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/893a986a-28a8-4f2e-9998-ec7cd7182c10-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:07:24.501466 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.501419    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/893a986a-28a8-4f2e-9998-ec7cd7182c10-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "893a986a-28a8-4f2e-9998-ec7cd7182c10" (UID: "893a986a-28a8-4f2e-9998-ec7cd7182c10"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:07:24.502951 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.502927    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/893a986a-28a8-4f2e-9998-ec7cd7182c10-kube-api-access-hp77h" (OuterVolumeSpecName: "kube-api-access-hp77h") pod "893a986a-28a8-4f2e-9998-ec7cd7182c10" (UID: "893a986a-28a8-4f2e-9998-ec7cd7182c10"). InnerVolumeSpecName "kube-api-access-hp77h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:07:24.503049 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.502971    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/893a986a-28a8-4f2e-9998-ec7cd7182c10-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "893a986a-28a8-4f2e-9998-ec7cd7182c10" (UID: "893a986a-28a8-4f2e-9998-ec7cd7182c10"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:07:24.602172 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.602139    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/893a986a-28a8-4f2e-9998-ec7cd7182c10-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:07:24.602172 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.602172    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp77h\" (UniqueName: \"kubernetes.io/projected/893a986a-28a8-4f2e-9998-ec7cd7182c10-kube-api-access-hp77h\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:07:24.602392 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.602186    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/893a986a-28a8-4f2e-9998-ec7cd7182c10-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:07:24.602392 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.602199    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/893a986a-28a8-4f2e-9998-ec7cd7182c10-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:07:24.805624 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.805534    2572 generic.go:356] "Generic (PLEG): container finished" podID="893a986a-28a8-4f2e-9998-ec7cd7182c10" containerID="5d1bf137c266f4d9b7f27c2550a8e445c75d57ec64ad343a116935665163255e" exitCode=0
Jun 15 06:07:24.805624 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.805600    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt" event={"ID":"893a986a-28a8-4f2e-9998-ec7cd7182c10","Type":"ContainerDied","Data":"5d1bf137c266f4d9b7f27c2550a8e445c75d57ec64ad343a116935665163255e"}
Jun 15 06:07:24.805833 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.805646    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt" event={"ID":"893a986a-28a8-4f2e-9998-ec7cd7182c10","Type":"ContainerDied","Data":"75c55ed374dd91398e592c7b7ce88898ca5003ff0c85ef1a0c6828a12873b68c"}
Jun 15 06:07:24.805833 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.805672    2572 scope.go:117] "RemoveContainer" containerID="b952be65a451562bdd989584438ab284d0e805cd5ff68938b0dd1ffd1c89892e"
Jun 15 06:07:24.805833 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.805599    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"
Jun 15 06:07:24.815079 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.815060    2572 scope.go:117] "RemoveContainer" containerID="5d1bf137c266f4d9b7f27c2550a8e445c75d57ec64ad343a116935665163255e"
Jun 15 06:07:24.822657 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.822637    2572 scope.go:117] "RemoveContainer" containerID="c080cc4bd45788fc415ea3bf505a1c6e1202978958632c87486422fe36b45ec8"
Jun 15 06:07:24.830849 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.830828    2572 scope.go:117] "RemoveContainer" containerID="b952be65a451562bdd989584438ab284d0e805cd5ff68938b0dd1ffd1c89892e"
Jun 15 06:07:24.831111 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.831087    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"]
Jun 15 06:07:24.831111 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:07:24.831101    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b952be65a451562bdd989584438ab284d0e805cd5ff68938b0dd1ffd1c89892e\": container with ID starting with b952be65a451562bdd989584438ab284d0e805cd5ff68938b0dd1ffd1c89892e not found: ID does not exist" containerID="b952be65a451562bdd989584438ab284d0e805cd5ff68938b0dd1ffd1c89892e"
Jun 15 06:07:24.831240 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.831132    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b952be65a451562bdd989584438ab284d0e805cd5ff68938b0dd1ffd1c89892e"} err="failed to get container status \"b952be65a451562bdd989584438ab284d0e805cd5ff68938b0dd1ffd1c89892e\": rpc error: code = NotFound desc = could not find container \"b952be65a451562bdd989584438ab284d0e805cd5ff68938b0dd1ffd1c89892e\": container with ID starting with b952be65a451562bdd989584438ab284d0e805cd5ff68938b0dd1ffd1c89892e not found: ID does not exist"
Jun 15 06:07:24.831240 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.831151    2572 scope.go:117] "RemoveContainer" containerID="5d1bf137c266f4d9b7f27c2550a8e445c75d57ec64ad343a116935665163255e"
Jun 15 06:07:24.831474 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:07:24.831452    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d1bf137c266f4d9b7f27c2550a8e445c75d57ec64ad343a116935665163255e\": container with ID starting with 5d1bf137c266f4d9b7f27c2550a8e445c75d57ec64ad343a116935665163255e not found: ID does not exist" containerID="5d1bf137c266f4d9b7f27c2550a8e445c75d57ec64ad343a116935665163255e"
Jun 15 06:07:24.831567 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.831482    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d1bf137c266f4d9b7f27c2550a8e445c75d57ec64ad343a116935665163255e"} err="failed to get container status \"5d1bf137c266f4d9b7f27c2550a8e445c75d57ec64ad343a116935665163255e\": rpc error: code = NotFound desc = could not find container \"5d1bf137c266f4d9b7f27c2550a8e445c75d57ec64ad343a116935665163255e\": container with ID starting with 5d1bf137c266f4d9b7f27c2550a8e445c75d57ec64ad343a116935665163255e not found: ID does not exist"
Jun 15 06:07:24.831567 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.831500    2572 scope.go:117] "RemoveContainer" containerID="c080cc4bd45788fc415ea3bf505a1c6e1202978958632c87486422fe36b45ec8"
Jun 15 06:07:24.831742 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:07:24.831726    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c080cc4bd45788fc415ea3bf505a1c6e1202978958632c87486422fe36b45ec8\": container with ID starting with c080cc4bd45788fc415ea3bf505a1c6e1202978958632c87486422fe36b45ec8 not found: ID does not exist" containerID="c080cc4bd45788fc415ea3bf505a1c6e1202978958632c87486422fe36b45ec8"
Jun 15 06:07:24.831780 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.831756    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c080cc4bd45788fc415ea3bf505a1c6e1202978958632c87486422fe36b45ec8"} err="failed to get container status \"c080cc4bd45788fc415ea3bf505a1c6e1202978958632c87486422fe36b45ec8\": rpc error: code = NotFound desc = could not find container \"c080cc4bd45788fc415ea3bf505a1c6e1202978958632c87486422fe36b45ec8\": container with ID starting with c080cc4bd45788fc415ea3bf505a1c6e1202978958632c87486422fe36b45ec8 not found: ID does not exist"
Jun 15 06:07:24.837830 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:24.837807    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-x29rt"]
Jun 15 06:07:25.323880 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:25.323849    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="893a986a-28a8-4f2e-9998-ec7cd7182c10" path="/var/lib/kubelet/pods/893a986a-28a8-4f2e-9998-ec7cd7182c10/volumes"
Jun 15 06:07:25.812618 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:25.812586    2572 generic.go:356] "Generic (PLEG): container finished" podID="3240d84a-03d9-45cc-98f3-66ef7e6f8381" containerID="e5c539963a2abcc81cf296bff7fd8168cfc3f441305acf62c7e348928b826e51" exitCode=0
Jun 15 06:07:25.812992 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:25.812656    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g" event={"ID":"3240d84a-03d9-45cc-98f3-66ef7e6f8381","Type":"ContainerDied","Data":"e5c539963a2abcc81cf296bff7fd8168cfc3f441305acf62c7e348928b826e51"}
Jun 15 06:07:25.813829 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:07:25.813813    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 15 06:08:05.361001 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:08:05.360975    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 06:08:05.365882 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:08:05.365858    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 06:09:53.613789 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:09:53.613749    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g" event={"ID":"3240d84a-03d9-45cc-98f3-66ef7e6f8381","Type":"ContainerStarted","Data":"4e4bd60541dc6644dc302ae9653390c0521851e933779cc7bb00af4995831dd4"}
Jun 15 06:09:53.614236 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:09:53.613822    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g" event={"ID":"3240d84a-03d9-45cc-98f3-66ef7e6f8381","Type":"ContainerStarted","Data":"db81161aa1b5fafb5a23be2d9eed655ab4db027d7e0bb243e42f6a954e21ef51"}
Jun 15 06:09:53.614236 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:09:53.613904    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"
Jun 15 06:09:53.643545 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:09:53.643469    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g" podStartSLOduration=3.761858924 podStartE2EDuration="2m34.643446324s" totalImagesPullingTime="2m26.953528872s" totalInitContainerRuntime="3.928058528s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:07:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:09:53.642667281 +0000 UTC m=+1308.790847271" watchObservedRunningTime="2026-06-15 06:09:53.643446324 +0000 UTC m=+1308.791626323"
Jun 15 06:09:54.619094 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:09:54.619064    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"
Jun 15 06:10:00.629977 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:00.629945    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"
Jun 15 06:10:30.634432 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:30.634344    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"
Jun 15 06:10:40.105820 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.105782    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"]
Jun 15 06:10:40.106284 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.106125    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g" podUID="3240d84a-03d9-45cc-98f3-66ef7e6f8381" containerName="kserve-container" containerID="cri-o://db81161aa1b5fafb5a23be2d9eed655ab4db027d7e0bb243e42f6a954e21ef51" gracePeriod=30
Jun 15 06:10:40.106284 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.106167    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g" podUID="3240d84a-03d9-45cc-98f3-66ef7e6f8381" containerName="kube-rbac-proxy" containerID="cri-o://4e4bd60541dc6644dc302ae9653390c0521851e933779cc7bb00af4995831dd4" gracePeriod=30
Jun 15 06:10:40.203532 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.203493    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"]
Jun 15 06:10:40.203983 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.203961    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="893a986a-28a8-4f2e-9998-ec7cd7182c10" containerName="kserve-container"
Jun 15 06:10:40.203983 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.203984    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="893a986a-28a8-4f2e-9998-ec7cd7182c10" containerName="kserve-container"
Jun 15 06:10:40.204116 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.203995    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="893a986a-28a8-4f2e-9998-ec7cd7182c10" containerName="storage-initializer"
Jun 15 06:10:40.204116 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.204001    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="893a986a-28a8-4f2e-9998-ec7cd7182c10" containerName="storage-initializer"
Jun 15 06:10:40.204116 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.204010    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="893a986a-28a8-4f2e-9998-ec7cd7182c10" containerName="kube-rbac-proxy"
Jun 15 06:10:40.204116 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.204016    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="893a986a-28a8-4f2e-9998-ec7cd7182c10" containerName="kube-rbac-proxy"
Jun 15 06:10:40.204116 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.204076    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="893a986a-28a8-4f2e-9998-ec7cd7182c10" containerName="kserve-container"
Jun 15 06:10:40.204116 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.204089    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="893a986a-28a8-4f2e-9998-ec7cd7182c10" containerName="kube-rbac-proxy"
Jun 15 06:10:40.206454 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.206439    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"
Jun 15 06:10:40.208660 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.208637    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Jun 15 06:10:40.208770 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.208710    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Jun 15 06:10:40.216987 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.216911    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"]
Jun 15 06:10:40.305030 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.304986    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/058a371b-dd05-4591-ba5d-271dfa9244c1-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv\" (UID: \"058a371b-dd05-4591-ba5d-271dfa9244c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"
Jun 15 06:10:40.305030 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.305028    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/058a371b-dd05-4591-ba5d-271dfa9244c1-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv\" (UID: \"058a371b-dd05-4591-ba5d-271dfa9244c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"
Jun 15 06:10:40.305308 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.305094    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqxxm\" (UniqueName: \"kubernetes.io/projected/058a371b-dd05-4591-ba5d-271dfa9244c1-kube-api-access-vqxxm\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv\" (UID: \"058a371b-dd05-4591-ba5d-271dfa9244c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"
Jun 15 06:10:40.305308 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.305136    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/058a371b-dd05-4591-ba5d-271dfa9244c1-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv\" (UID: \"058a371b-dd05-4591-ba5d-271dfa9244c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"
Jun 15 06:10:40.305308 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.305160    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/058a371b-dd05-4591-ba5d-271dfa9244c1-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv\" (UID: \"058a371b-dd05-4591-ba5d-271dfa9244c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"
Jun 15 06:10:40.406492 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.406436    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/058a371b-dd05-4591-ba5d-271dfa9244c1-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv\" (UID: \"058a371b-dd05-4591-ba5d-271dfa9244c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"
Jun 15 06:10:40.406492 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.406488    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/058a371b-dd05-4591-ba5d-271dfa9244c1-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv\" (UID: \"058a371b-dd05-4591-ba5d-271dfa9244c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"
Jun 15 06:10:40.406792 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.406536    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqxxm\" (UniqueName: \"kubernetes.io/projected/058a371b-dd05-4591-ba5d-271dfa9244c1-kube-api-access-vqxxm\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv\" (UID: \"058a371b-dd05-4591-ba5d-271dfa9244c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"
Jun 15 06:10:40.406792 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.406575    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/058a371b-dd05-4591-ba5d-271dfa9244c1-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv\" (UID: \"058a371b-dd05-4591-ba5d-271dfa9244c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"
Jun 15 06:10:40.406792 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.406594    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/058a371b-dd05-4591-ba5d-271dfa9244c1-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv\" (UID: \"058a371b-dd05-4591-ba5d-271dfa9244c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"
Jun 15 06:10:40.406792 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:10:40.406682    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-serving-cert: secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
Jun 15 06:10:40.406792 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:10:40.406768    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/058a371b-dd05-4591-ba5d-271dfa9244c1-proxy-tls podName:058a371b-dd05-4591-ba5d-271dfa9244c1 nodeName:}" failed. No retries permitted until 2026-06-15 06:10:40.906750454 +0000 UTC m=+1356.054930439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/058a371b-dd05-4591-ba5d-271dfa9244c1-proxy-tls") pod "isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv" (UID: "058a371b-dd05-4591-ba5d-271dfa9244c1") : secret "isvc-lightgbm-v2-kserve-predictor-serving-cert" not found
Jun 15 06:10:40.407034 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.406991    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/058a371b-dd05-4591-ba5d-271dfa9244c1-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv\" (UID: \"058a371b-dd05-4591-ba5d-271dfa9244c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"
Jun 15 06:10:40.407292 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.407267    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/058a371b-dd05-4591-ba5d-271dfa9244c1-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv\" (UID: \"058a371b-dd05-4591-ba5d-271dfa9244c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"
Jun 15 06:10:40.407423 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.407267    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/058a371b-dd05-4591-ba5d-271dfa9244c1-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv\" (UID: \"058a371b-dd05-4591-ba5d-271dfa9244c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"
Jun 15 06:10:40.417767 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.417740    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqxxm\" (UniqueName: \"kubernetes.io/projected/058a371b-dd05-4591-ba5d-271dfa9244c1-kube-api-access-vqxxm\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv\" (UID: \"058a371b-dd05-4591-ba5d-271dfa9244c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"
Jun 15 06:10:40.624295 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.624251    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g" podUID="3240d84a-03d9-45cc-98f3-66ef7e6f8381" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Jun 15 06:10:40.630774 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.630737    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g" podUID="3240d84a-03d9-45cc-98f3-66ef7e6f8381" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.35:8080/v2/models/isvc-lightgbm-v2-runtime/ready\": dial tcp 10.134.0.35:8080: connect: connection refused"
Jun 15 06:10:40.825267 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.825232    2572 generic.go:356] "Generic (PLEG): container finished" podID="3240d84a-03d9-45cc-98f3-66ef7e6f8381" containerID="4e4bd60541dc6644dc302ae9653390c0521851e933779cc7bb00af4995831dd4" exitCode=2
Jun 15 06:10:40.825467 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.825302    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g" event={"ID":"3240d84a-03d9-45cc-98f3-66ef7e6f8381","Type":"ContainerDied","Data":"4e4bd60541dc6644dc302ae9653390c0521851e933779cc7bb00af4995831dd4"}
Jun 15 06:10:40.911078 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.911046    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/058a371b-dd05-4591-ba5d-271dfa9244c1-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv\" (UID: \"058a371b-dd05-4591-ba5d-271dfa9244c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"
Jun 15 06:10:40.913499 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:40.913475    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/058a371b-dd05-4591-ba5d-271dfa9244c1-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv\" (UID: \"058a371b-dd05-4591-ba5d-271dfa9244c1\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"
Jun 15 06:10:41.118263 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.118229    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"
Jun 15 06:10:41.154484 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.154461    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"
Jun 15 06:10:41.258812 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.254147    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"]
Jun 15 06:10:41.262171 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:10:41.262124    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod058a371b_dd05_4591_ba5d_271dfa9244c1.slice/crio-3dba7a65ca8881f64bf2db3b9291527df9648a9f599b3d73307cdece1162e3d9 WatchSource:0}: Error finding container 3dba7a65ca8881f64bf2db3b9291527df9648a9f599b3d73307cdece1162e3d9: Status 404 returned error can't find the container with id 3dba7a65ca8881f64bf2db3b9291527df9648a9f599b3d73307cdece1162e3d9
Jun 15 06:10:41.313672 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.313647    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3240d84a-03d9-45cc-98f3-66ef7e6f8381-cabundle-cert\") pod \"3240d84a-03d9-45cc-98f3-66ef7e6f8381\" (UID: \"3240d84a-03d9-45cc-98f3-66ef7e6f8381\") "
Jun 15 06:10:41.313795 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.313683    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3240d84a-03d9-45cc-98f3-66ef7e6f8381-kserve-provision-location\") pod \"3240d84a-03d9-45cc-98f3-66ef7e6f8381\" (UID: \"3240d84a-03d9-45cc-98f3-66ef7e6f8381\") "
Jun 15 06:10:41.313795 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.313711    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3240d84a-03d9-45cc-98f3-66ef7e6f8381-kube-rbac-proxy-sar-config\") pod \"3240d84a-03d9-45cc-98f3-66ef7e6f8381\" (UID: \"3240d84a-03d9-45cc-98f3-66ef7e6f8381\") "
Jun 15 06:10:41.313795 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.313744    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3240d84a-03d9-45cc-98f3-66ef7e6f8381-proxy-tls\") pod \"3240d84a-03d9-45cc-98f3-66ef7e6f8381\" (UID: \"3240d84a-03d9-45cc-98f3-66ef7e6f8381\") "
Jun 15 06:10:41.313966 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.313805    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnnl8\" (UniqueName: \"kubernetes.io/projected/3240d84a-03d9-45cc-98f3-66ef7e6f8381-kube-api-access-gnnl8\") pod \"3240d84a-03d9-45cc-98f3-66ef7e6f8381\" (UID: \"3240d84a-03d9-45cc-98f3-66ef7e6f8381\") "
Jun 15 06:10:41.314052 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.314026    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3240d84a-03d9-45cc-98f3-66ef7e6f8381-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3240d84a-03d9-45cc-98f3-66ef7e6f8381" (UID: "3240d84a-03d9-45cc-98f3-66ef7e6f8381"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:10:41.314124 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.314044    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3240d84a-03d9-45cc-98f3-66ef7e6f8381-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3240d84a-03d9-45cc-98f3-66ef7e6f8381" (UID: "3240d84a-03d9-45cc-98f3-66ef7e6f8381"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:10:41.314239 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.314129    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3240d84a-03d9-45cc-98f3-66ef7e6f8381-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:10:41.314239 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.314144    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3240d84a-03d9-45cc-98f3-66ef7e6f8381-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:10:41.314341 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.314296    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3240d84a-03d9-45cc-98f3-66ef7e6f8381-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3240d84a-03d9-45cc-98f3-66ef7e6f8381" (UID: "3240d84a-03d9-45cc-98f3-66ef7e6f8381"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:10:41.315958 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.315926    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3240d84a-03d9-45cc-98f3-66ef7e6f8381-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3240d84a-03d9-45cc-98f3-66ef7e6f8381" (UID: "3240d84a-03d9-45cc-98f3-66ef7e6f8381"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:10:41.316065 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.316036    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3240d84a-03d9-45cc-98f3-66ef7e6f8381-kube-api-access-gnnl8" (OuterVolumeSpecName: "kube-api-access-gnnl8") pod "3240d84a-03d9-45cc-98f3-66ef7e6f8381" (UID: "3240d84a-03d9-45cc-98f3-66ef7e6f8381"). InnerVolumeSpecName "kube-api-access-gnnl8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:10:41.415103 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.415075    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnnl8\" (UniqueName: \"kubernetes.io/projected/3240d84a-03d9-45cc-98f3-66ef7e6f8381-kube-api-access-gnnl8\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:10:41.415103 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.415100    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3240d84a-03d9-45cc-98f3-66ef7e6f8381-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:10:41.415103 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.415109    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3240d84a-03d9-45cc-98f3-66ef7e6f8381-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:10:41.833958 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.833917    2572 generic.go:356] "Generic (PLEG): container finished" podID="3240d84a-03d9-45cc-98f3-66ef7e6f8381" containerID="db81161aa1b5fafb5a23be2d9eed655ab4db027d7e0bb243e42f6a954e21ef51" exitCode=0
Jun 15 06:10:41.834167 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.834002    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"
Jun 15 06:10:41.834167 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.834001    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g" event={"ID":"3240d84a-03d9-45cc-98f3-66ef7e6f8381","Type":"ContainerDied","Data":"db81161aa1b5fafb5a23be2d9eed655ab4db027d7e0bb243e42f6a954e21ef51"}
Jun 15 06:10:41.834167 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.834075    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g" event={"ID":"3240d84a-03d9-45cc-98f3-66ef7e6f8381","Type":"ContainerDied","Data":"331c428f88c0c77e8ec3e6c5dbdb5c5cf53985d60ee2a41817e3e2fe77fdf21e"}
Jun 15 06:10:41.834167 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.834109    2572 scope.go:117] "RemoveContainer" containerID="4e4bd60541dc6644dc302ae9653390c0521851e933779cc7bb00af4995831dd4"
Jun 15 06:10:41.835902 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.835877    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv" event={"ID":"058a371b-dd05-4591-ba5d-271dfa9244c1","Type":"ContainerStarted","Data":"59c53900110943060849329c80cf96d982990bbbba24bebadeca95311266100c"}
Jun 15 06:10:41.836036 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.835928    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv" event={"ID":"058a371b-dd05-4591-ba5d-271dfa9244c1","Type":"ContainerStarted","Data":"3dba7a65ca8881f64bf2db3b9291527df9648a9f599b3d73307cdece1162e3d9"}
Jun 15 06:10:41.843178 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.843136    2572 scope.go:117] "RemoveContainer" containerID="db81161aa1b5fafb5a23be2d9eed655ab4db027d7e0bb243e42f6a954e21ef51"
Jun 15 06:10:41.853151 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.853112    2572 scope.go:117] "RemoveContainer" containerID="e5c539963a2abcc81cf296bff7fd8168cfc3f441305acf62c7e348928b826e51"
Jun 15 06:10:41.856275 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.856251    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"]
Jun 15 06:10:41.859842 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.859821    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-runtime-predictor-7f4957c5d5-5454g"]
Jun 15 06:10:41.863446 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.863382    2572 scope.go:117] "RemoveContainer" containerID="4e4bd60541dc6644dc302ae9653390c0521851e933779cc7bb00af4995831dd4"
Jun 15 06:10:41.863700 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:10:41.863680    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e4bd60541dc6644dc302ae9653390c0521851e933779cc7bb00af4995831dd4\": container with ID starting with 4e4bd60541dc6644dc302ae9653390c0521851e933779cc7bb00af4995831dd4 not found: ID does not exist" containerID="4e4bd60541dc6644dc302ae9653390c0521851e933779cc7bb00af4995831dd4"
Jun 15 06:10:41.863772 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.863712    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e4bd60541dc6644dc302ae9653390c0521851e933779cc7bb00af4995831dd4"} err="failed to get container status \"4e4bd60541dc6644dc302ae9653390c0521851e933779cc7bb00af4995831dd4\": rpc error: code = NotFound desc = could not find container \"4e4bd60541dc6644dc302ae9653390c0521851e933779cc7bb00af4995831dd4\": container with ID starting with 4e4bd60541dc6644dc302ae9653390c0521851e933779cc7bb00af4995831dd4 not found: ID does not exist"
Jun 15 06:10:41.863772 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.863737    2572 scope.go:117] "RemoveContainer" containerID="db81161aa1b5fafb5a23be2d9eed655ab4db027d7e0bb243e42f6a954e21ef51"
Jun 15 06:10:41.864020 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:10:41.864003    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db81161aa1b5fafb5a23be2d9eed655ab4db027d7e0bb243e42f6a954e21ef51\": container with ID starting with db81161aa1b5fafb5a23be2d9eed655ab4db027d7e0bb243e42f6a954e21ef51 not found: ID does not exist" containerID="db81161aa1b5fafb5a23be2d9eed655ab4db027d7e0bb243e42f6a954e21ef51"
Jun 15 06:10:41.864065 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.864026    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db81161aa1b5fafb5a23be2d9eed655ab4db027d7e0bb243e42f6a954e21ef51"} err="failed to get container status \"db81161aa1b5fafb5a23be2d9eed655ab4db027d7e0bb243e42f6a954e21ef51\": rpc error: code = NotFound desc = could not find container \"db81161aa1b5fafb5a23be2d9eed655ab4db027d7e0bb243e42f6a954e21ef51\": container with ID starting with db81161aa1b5fafb5a23be2d9eed655ab4db027d7e0bb243e42f6a954e21ef51 not found: ID does not exist"
Jun 15 06:10:41.864065 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.864041    2572 scope.go:117] "RemoveContainer" containerID="e5c539963a2abcc81cf296bff7fd8168cfc3f441305acf62c7e348928b826e51"
Jun 15 06:10:41.864304 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:10:41.864288    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5c539963a2abcc81cf296bff7fd8168cfc3f441305acf62c7e348928b826e51\": container with ID starting with e5c539963a2abcc81cf296bff7fd8168cfc3f441305acf62c7e348928b826e51 not found: ID does not exist" containerID="e5c539963a2abcc81cf296bff7fd8168cfc3f441305acf62c7e348928b826e51"
Jun 15 06:10:41.864350 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:41.864311    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5c539963a2abcc81cf296bff7fd8168cfc3f441305acf62c7e348928b826e51"} err="failed to get container status \"e5c539963a2abcc81cf296bff7fd8168cfc3f441305acf62c7e348928b826e51\": rpc error: code = NotFound desc = could not find container \"e5c539963a2abcc81cf296bff7fd8168cfc3f441305acf62c7e348928b826e51\": container with ID starting with e5c539963a2abcc81cf296bff7fd8168cfc3f441305acf62c7e348928b826e51 not found: ID does not exist"
Jun 15 06:10:43.323961 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:43.323928    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3240d84a-03d9-45cc-98f3-66ef7e6f8381" path="/var/lib/kubelet/pods/3240d84a-03d9-45cc-98f3-66ef7e6f8381/volumes"
Jun 15 06:10:45.857445 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:45.857415    2572 generic.go:356] "Generic (PLEG): container finished" podID="058a371b-dd05-4591-ba5d-271dfa9244c1" containerID="59c53900110943060849329c80cf96d982990bbbba24bebadeca95311266100c" exitCode=0
Jun 15 06:10:45.857827 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:45.857491    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv" event={"ID":"058a371b-dd05-4591-ba5d-271dfa9244c1","Type":"ContainerDied","Data":"59c53900110943060849329c80cf96d982990bbbba24bebadeca95311266100c"}
Jun 15 06:10:46.864843 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:46.864807    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv" event={"ID":"058a371b-dd05-4591-ba5d-271dfa9244c1","Type":"ContainerStarted","Data":"a21a0c85cc46d9fee55b6fcc323344fe7aa82c76a62a0d2c68b22b27cc40b7f5"}
Jun 15 06:10:46.865316 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:46.864882    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv" event={"ID":"058a371b-dd05-4591-ba5d-271dfa9244c1","Type":"ContainerStarted","Data":"cf6e41ecdd055a87a58b46efb4c813bcb0298790824aec70fc41d75681d8a8db"}
Jun 15 06:10:46.865316 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:46.865105    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"
Jun 15 06:10:46.896615 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:46.896558    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv" podStartSLOduration=2.909114931 podStartE2EDuration="6.896541502s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.987426571s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:10:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:10:46.893691292 +0000 UTC m=+1362.041871289" watchObservedRunningTime="2026-06-15 06:10:46.896541502 +0000 UTC m=+1362.044721559"
Jun 15 06:10:47.869737 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:47.869699    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"
Jun 15 06:10:47.871126 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:47.871102    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv" podUID="058a371b-dd05-4591-ba5d-271dfa9244c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jun 15 06:10:48.875393 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:48.875342    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv" podUID="058a371b-dd05-4591-ba5d-271dfa9244c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jun 15 06:10:53.880333 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:53.880307    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"
Jun 15 06:10:53.881371 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:10:53.881340    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"
Jun 15 06:11:00.271215 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.271177    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"]
Jun 15 06:11:00.271652 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.271437    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv" podUID="058a371b-dd05-4591-ba5d-271dfa9244c1" containerName="kserve-container" containerID="cri-o://cf6e41ecdd055a87a58b46efb4c813bcb0298790824aec70fc41d75681d8a8db" gracePeriod=30
Jun 15 06:11:00.271652 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.271460    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv" podUID="058a371b-dd05-4591-ba5d-271dfa9244c1" containerName="kube-rbac-proxy" containerID="cri-o://a21a0c85cc46d9fee55b6fcc323344fe7aa82c76a62a0d2c68b22b27cc40b7f5" gracePeriod=30
Jun 15 06:11:00.441188 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.441158    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"]
Jun 15 06:11:00.441624 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.441606    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3240d84a-03d9-45cc-98f3-66ef7e6f8381" containerName="storage-initializer"
Jun 15 06:11:00.441712 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.441627    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3240d84a-03d9-45cc-98f3-66ef7e6f8381" containerName="storage-initializer"
Jun 15 06:11:00.441712 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.441647    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3240d84a-03d9-45cc-98f3-66ef7e6f8381" containerName="kube-rbac-proxy"
Jun 15 06:11:00.441712 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.441656    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3240d84a-03d9-45cc-98f3-66ef7e6f8381" containerName="kube-rbac-proxy"
Jun 15 06:11:00.441712 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.441668    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3240d84a-03d9-45cc-98f3-66ef7e6f8381" containerName="kserve-container"
Jun 15 06:11:00.441712 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.441677    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3240d84a-03d9-45cc-98f3-66ef7e6f8381" containerName="kserve-container"
Jun 15 06:11:00.441963 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.441773    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3240d84a-03d9-45cc-98f3-66ef7e6f8381" containerName="kserve-container"
Jun 15 06:11:00.441963 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.441783    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3240d84a-03d9-45cc-98f3-66ef7e6f8381" containerName="kube-rbac-proxy"
Jun 15 06:11:00.444032 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.444011    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"
Jun 15 06:11:00.449767 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.449751    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-predictor-serving-cert\""
Jun 15 06:11:00.449864 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.449771    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-mlflow-v2-runtime-kube-rbac-proxy-sar-config\""
Jun 15 06:11:00.467400 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.467351    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"]
Jun 15 06:11:00.486342 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.486308    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c64dc4b-a424-4464-9e45-253577db1e35-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5\" (UID: \"5c64dc4b-a424-4464-9e45-253577db1e35\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"
Jun 15 06:11:00.486463 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.486427    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c64dc4b-a424-4464-9e45-253577db1e35-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5\" (UID: \"5c64dc4b-a424-4464-9e45-253577db1e35\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"
Jun 15 06:11:00.486529 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.486481    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c64dc4b-a424-4464-9e45-253577db1e35-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5\" (UID: \"5c64dc4b-a424-4464-9e45-253577db1e35\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"
Jun 15 06:11:00.486529 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.486504    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drgzn\" (UniqueName: \"kubernetes.io/projected/5c64dc4b-a424-4464-9e45-253577db1e35-kube-api-access-drgzn\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5\" (UID: \"5c64dc4b-a424-4464-9e45-253577db1e35\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"
Jun 15 06:11:00.486652 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.486537    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c64dc4b-a424-4464-9e45-253577db1e35-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5\" (UID: \"5c64dc4b-a424-4464-9e45-253577db1e35\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"
Jun 15 06:11:00.587954 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.587868    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c64dc4b-a424-4464-9e45-253577db1e35-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5\" (UID: \"5c64dc4b-a424-4464-9e45-253577db1e35\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"
Jun 15 06:11:00.587954 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.587931    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c64dc4b-a424-4464-9e45-253577db1e35-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5\" (UID: \"5c64dc4b-a424-4464-9e45-253577db1e35\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"
Jun 15 06:11:00.588186 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.588029    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drgzn\" (UniqueName: \"kubernetes.io/projected/5c64dc4b-a424-4464-9e45-253577db1e35-kube-api-access-drgzn\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5\" (UID: \"5c64dc4b-a424-4464-9e45-253577db1e35\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"
Jun 15 06:11:00.588186 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:11:00.588032    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-serving-cert: secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Jun 15 06:11:00.588186 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.588059    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c64dc4b-a424-4464-9e45-253577db1e35-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5\" (UID: \"5c64dc4b-a424-4464-9e45-253577db1e35\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"
Jun 15 06:11:00.588186 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:11:00.588120    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c64dc4b-a424-4464-9e45-253577db1e35-proxy-tls podName:5c64dc4b-a424-4464-9e45-253577db1e35 nodeName:}" failed. No retries permitted until 2026-06-15 06:11:01.088096428 +0000 UTC m=+1376.236276404 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5c64dc4b-a424-4464-9e45-253577db1e35-proxy-tls") pod "isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5" (UID: "5c64dc4b-a424-4464-9e45-253577db1e35") : secret "isvc-mlflow-v2-runtime-predictor-serving-cert" not found
Jun 15 06:11:00.588186 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.588142    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c64dc4b-a424-4464-9e45-253577db1e35-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5\" (UID: \"5c64dc4b-a424-4464-9e45-253577db1e35\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"
Jun 15 06:11:00.588507 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.588483    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c64dc4b-a424-4464-9e45-253577db1e35-kserve-provision-location\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5\" (UID: \"5c64dc4b-a424-4464-9e45-253577db1e35\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"
Jun 15 06:11:00.588757 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.588735    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c64dc4b-a424-4464-9e45-253577db1e35-kube-rbac-proxy-sar-config\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5\" (UID: \"5c64dc4b-a424-4464-9e45-253577db1e35\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"
Jun 15 06:11:00.588836 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.588765    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c64dc4b-a424-4464-9e45-253577db1e35-cabundle-cert\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5\" (UID: \"5c64dc4b-a424-4464-9e45-253577db1e35\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"
Jun 15 06:11:00.605024 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.604995    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drgzn\" (UniqueName: \"kubernetes.io/projected/5c64dc4b-a424-4464-9e45-253577db1e35-kube-api-access-drgzn\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5\" (UID: \"5c64dc4b-a424-4464-9e45-253577db1e35\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"
Jun 15 06:11:00.923548 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.923523    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"
Jun 15 06:11:00.931712 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.931688    2572 generic.go:356] "Generic (PLEG): container finished" podID="058a371b-dd05-4591-ba5d-271dfa9244c1" containerID="a21a0c85cc46d9fee55b6fcc323344fe7aa82c76a62a0d2c68b22b27cc40b7f5" exitCode=2
Jun 15 06:11:00.931712 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.931711    2572 generic.go:356] "Generic (PLEG): container finished" podID="058a371b-dd05-4591-ba5d-271dfa9244c1" containerID="cf6e41ecdd055a87a58b46efb4c813bcb0298790824aec70fc41d75681d8a8db" exitCode=0
Jun 15 06:11:00.931858 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.931740    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"
Jun 15 06:11:00.931858 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.931743    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv" event={"ID":"058a371b-dd05-4591-ba5d-271dfa9244c1","Type":"ContainerDied","Data":"a21a0c85cc46d9fee55b6fcc323344fe7aa82c76a62a0d2c68b22b27cc40b7f5"}
Jun 15 06:11:00.931919 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.931875    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv" event={"ID":"058a371b-dd05-4591-ba5d-271dfa9244c1","Type":"ContainerDied","Data":"cf6e41ecdd055a87a58b46efb4c813bcb0298790824aec70fc41d75681d8a8db"}
Jun 15 06:11:00.931919 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.931910    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv" event={"ID":"058a371b-dd05-4591-ba5d-271dfa9244c1","Type":"ContainerDied","Data":"3dba7a65ca8881f64bf2db3b9291527df9648a9f599b3d73307cdece1162e3d9"}
Jun 15 06:11:00.931986 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.931924    2572 scope.go:117] "RemoveContainer" containerID="a21a0c85cc46d9fee55b6fcc323344fe7aa82c76a62a0d2c68b22b27cc40b7f5"
Jun 15 06:11:00.940477 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.940445    2572 scope.go:117] "RemoveContainer" containerID="cf6e41ecdd055a87a58b46efb4c813bcb0298790824aec70fc41d75681d8a8db"
Jun 15 06:11:00.947758 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.947740    2572 scope.go:117] "RemoveContainer" containerID="59c53900110943060849329c80cf96d982990bbbba24bebadeca95311266100c"
Jun 15 06:11:00.956873 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.956852    2572 scope.go:117] "RemoveContainer" containerID="a21a0c85cc46d9fee55b6fcc323344fe7aa82c76a62a0d2c68b22b27cc40b7f5"
Jun 15 06:11:00.957149 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:11:00.957127    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a21a0c85cc46d9fee55b6fcc323344fe7aa82c76a62a0d2c68b22b27cc40b7f5\": container with ID starting with a21a0c85cc46d9fee55b6fcc323344fe7aa82c76a62a0d2c68b22b27cc40b7f5 not found: ID does not exist" containerID="a21a0c85cc46d9fee55b6fcc323344fe7aa82c76a62a0d2c68b22b27cc40b7f5"
Jun 15 06:11:00.957206 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.957158    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a21a0c85cc46d9fee55b6fcc323344fe7aa82c76a62a0d2c68b22b27cc40b7f5"} err="failed to get container status \"a21a0c85cc46d9fee55b6fcc323344fe7aa82c76a62a0d2c68b22b27cc40b7f5\": rpc error: code = NotFound desc = could not find container \"a21a0c85cc46d9fee55b6fcc323344fe7aa82c76a62a0d2c68b22b27cc40b7f5\": container with ID starting with a21a0c85cc46d9fee55b6fcc323344fe7aa82c76a62a0d2c68b22b27cc40b7f5 not found: ID does not exist"
Jun 15 06:11:00.957206 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.957174    2572 scope.go:117] "RemoveContainer" containerID="cf6e41ecdd055a87a58b46efb4c813bcb0298790824aec70fc41d75681d8a8db"
Jun 15 06:11:00.957444 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:11:00.957427    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf6e41ecdd055a87a58b46efb4c813bcb0298790824aec70fc41d75681d8a8db\": container with ID starting with cf6e41ecdd055a87a58b46efb4c813bcb0298790824aec70fc41d75681d8a8db not found: ID does not exist" containerID="cf6e41ecdd055a87a58b46efb4c813bcb0298790824aec70fc41d75681d8a8db"
Jun 15 06:11:00.957490 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.957452    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf6e41ecdd055a87a58b46efb4c813bcb0298790824aec70fc41d75681d8a8db"} err="failed to get container status \"cf6e41ecdd055a87a58b46efb4c813bcb0298790824aec70fc41d75681d8a8db\": rpc error: code = NotFound desc = could not find container \"cf6e41ecdd055a87a58b46efb4c813bcb0298790824aec70fc41d75681d8a8db\": container with ID starting with cf6e41ecdd055a87a58b46efb4c813bcb0298790824aec70fc41d75681d8a8db not found: ID does not exist"
Jun 15 06:11:00.957490 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.957468    2572 scope.go:117] "RemoveContainer" containerID="59c53900110943060849329c80cf96d982990bbbba24bebadeca95311266100c"
Jun 15 06:11:00.957712 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:11:00.957691    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59c53900110943060849329c80cf96d982990bbbba24bebadeca95311266100c\": container with ID starting with 59c53900110943060849329c80cf96d982990bbbba24bebadeca95311266100c not found: ID does not exist" containerID="59c53900110943060849329c80cf96d982990bbbba24bebadeca95311266100c"
Jun 15 06:11:00.957751 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.957722    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59c53900110943060849329c80cf96d982990bbbba24bebadeca95311266100c"} err="failed to get container status \"59c53900110943060849329c80cf96d982990bbbba24bebadeca95311266100c\": rpc error: code = NotFound desc = could not find container \"59c53900110943060849329c80cf96d982990bbbba24bebadeca95311266100c\": container with ID starting with 59c53900110943060849329c80cf96d982990bbbba24bebadeca95311266100c not found: ID does not exist"
Jun 15 06:11:00.957751 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.957746    2572 scope.go:117] "RemoveContainer" containerID="a21a0c85cc46d9fee55b6fcc323344fe7aa82c76a62a0d2c68b22b27cc40b7f5"
Jun 15 06:11:00.957979 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.957960    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a21a0c85cc46d9fee55b6fcc323344fe7aa82c76a62a0d2c68b22b27cc40b7f5"} err="failed to get container status \"a21a0c85cc46d9fee55b6fcc323344fe7aa82c76a62a0d2c68b22b27cc40b7f5\": rpc error: code = NotFound desc = could not find container \"a21a0c85cc46d9fee55b6fcc323344fe7aa82c76a62a0d2c68b22b27cc40b7f5\": container with ID starting with a21a0c85cc46d9fee55b6fcc323344fe7aa82c76a62a0d2c68b22b27cc40b7f5 not found: ID does not exist"
Jun 15 06:11:00.958023 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.957981    2572 scope.go:117] "RemoveContainer" containerID="cf6e41ecdd055a87a58b46efb4c813bcb0298790824aec70fc41d75681d8a8db"
Jun 15 06:11:00.958218 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.958197    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf6e41ecdd055a87a58b46efb4c813bcb0298790824aec70fc41d75681d8a8db"} err="failed to get container status \"cf6e41ecdd055a87a58b46efb4c813bcb0298790824aec70fc41d75681d8a8db\": rpc error: code = NotFound desc = could not find container \"cf6e41ecdd055a87a58b46efb4c813bcb0298790824aec70fc41d75681d8a8db\": container with ID starting with cf6e41ecdd055a87a58b46efb4c813bcb0298790824aec70fc41d75681d8a8db not found: ID does not exist"
Jun 15 06:11:00.958263 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.958221    2572 scope.go:117] "RemoveContainer" containerID="59c53900110943060849329c80cf96d982990bbbba24bebadeca95311266100c"
Jun 15 06:11:00.958577 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.958560    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59c53900110943060849329c80cf96d982990bbbba24bebadeca95311266100c"} err="failed to get container status \"59c53900110943060849329c80cf96d982990bbbba24bebadeca95311266100c\": rpc error: code = NotFound desc = could not find container \"59c53900110943060849329c80cf96d982990bbbba24bebadeca95311266100c\": container with ID starting with 59c53900110943060849329c80cf96d982990bbbba24bebadeca95311266100c not found: ID does not exist"
Jun 15 06:11:00.991673 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.991650    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/058a371b-dd05-4591-ba5d-271dfa9244c1-kserve-provision-location\") pod \"058a371b-dd05-4591-ba5d-271dfa9244c1\" (UID: \"058a371b-dd05-4591-ba5d-271dfa9244c1\") "
Jun 15 06:11:00.991767 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.991713    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/058a371b-dd05-4591-ba5d-271dfa9244c1-proxy-tls\") pod \"058a371b-dd05-4591-ba5d-271dfa9244c1\" (UID: \"058a371b-dd05-4591-ba5d-271dfa9244c1\") "
Jun 15 06:11:00.991767 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.991743    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/058a371b-dd05-4591-ba5d-271dfa9244c1-cabundle-cert\") pod \"058a371b-dd05-4591-ba5d-271dfa9244c1\" (UID: \"058a371b-dd05-4591-ba5d-271dfa9244c1\") "
Jun 15 06:11:00.991883 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.991766    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/058a371b-dd05-4591-ba5d-271dfa9244c1-kube-rbac-proxy-sar-config\") pod \"058a371b-dd05-4591-ba5d-271dfa9244c1\" (UID: \"058a371b-dd05-4591-ba5d-271dfa9244c1\") "
Jun 15 06:11:00.991883 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.991789    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vqxxm\" (UniqueName: \"kubernetes.io/projected/058a371b-dd05-4591-ba5d-271dfa9244c1-kube-api-access-vqxxm\") pod \"058a371b-dd05-4591-ba5d-271dfa9244c1\" (UID: \"058a371b-dd05-4591-ba5d-271dfa9244c1\") "
Jun 15 06:11:00.992046 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.992019    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/058a371b-dd05-4591-ba5d-271dfa9244c1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "058a371b-dd05-4591-ba5d-271dfa9244c1" (UID: "058a371b-dd05-4591-ba5d-271dfa9244c1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:11:00.992126 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.992109    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/058a371b-dd05-4591-ba5d-271dfa9244c1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "058a371b-dd05-4591-ba5d-271dfa9244c1" (UID: "058a371b-dd05-4591-ba5d-271dfa9244c1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:11:00.992175 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.992120    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/058a371b-dd05-4591-ba5d-271dfa9244c1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "058a371b-dd05-4591-ba5d-271dfa9244c1" (UID: "058a371b-dd05-4591-ba5d-271dfa9244c1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:11:00.993749 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.993729    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/058a371b-dd05-4591-ba5d-271dfa9244c1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "058a371b-dd05-4591-ba5d-271dfa9244c1" (UID: "058a371b-dd05-4591-ba5d-271dfa9244c1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:11:00.993818 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:00.993761    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/058a371b-dd05-4591-ba5d-271dfa9244c1-kube-api-access-vqxxm" (OuterVolumeSpecName: "kube-api-access-vqxxm") pod "058a371b-dd05-4591-ba5d-271dfa9244c1" (UID: "058a371b-dd05-4591-ba5d-271dfa9244c1"). InnerVolumeSpecName "kube-api-access-vqxxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:11:01.093008 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:01.092919    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c64dc4b-a424-4464-9e45-253577db1e35-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5\" (UID: \"5c64dc4b-a424-4464-9e45-253577db1e35\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"
Jun 15 06:11:01.093008 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:01.093008    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/058a371b-dd05-4591-ba5d-271dfa9244c1-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:11:01.093236 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:01.093022    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/058a371b-dd05-4591-ba5d-271dfa9244c1-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:11:01.093236 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:01.093032    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/058a371b-dd05-4591-ba5d-271dfa9244c1-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:11:01.093236 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:01.093062    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/058a371b-dd05-4591-ba5d-271dfa9244c1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:11:01.093236 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:01.093071    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqxxm\" (UniqueName: \"kubernetes.io/projected/058a371b-dd05-4591-ba5d-271dfa9244c1-kube-api-access-vqxxm\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:11:01.095452 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:01.095430    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c64dc4b-a424-4464-9e45-253577db1e35-proxy-tls\") pod \"isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5\" (UID: \"5c64dc4b-a424-4464-9e45-253577db1e35\") " pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"
Jun 15 06:11:01.257945 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:01.257913    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"]
Jun 15 06:11:01.264213 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:01.264189    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-b5lzv"]
Jun 15 06:11:01.324236 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:01.324200    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="058a371b-dd05-4591-ba5d-271dfa9244c1" path="/var/lib/kubelet/pods/058a371b-dd05-4591-ba5d-271dfa9244c1/volumes"
Jun 15 06:11:01.356345 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:01.356268    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"
Jun 15 06:11:01.505419 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:01.505385    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"]
Jun 15 06:11:01.508412 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:11:01.508381    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c64dc4b_a424_4464_9e45_253577db1e35.slice/crio-30e8a4376508f38a133a3efd412f43e1d89e74eb9f8b496b12f559e8e8462d87 WatchSource:0}: Error finding container 30e8a4376508f38a133a3efd412f43e1d89e74eb9f8b496b12f559e8e8462d87: Status 404 returned error can't find the container with id 30e8a4376508f38a133a3efd412f43e1d89e74eb9f8b496b12f559e8e8462d87
Jun 15 06:11:01.938950 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:01.938911    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5" event={"ID":"5c64dc4b-a424-4464-9e45-253577db1e35","Type":"ContainerStarted","Data":"a7b3c75bbe892905eb59b1793328c316484eea23cdf5485c09055161431ba1da"}
Jun 15 06:11:01.939166 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:01.938969    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5" event={"ID":"5c64dc4b-a424-4464-9e45-253577db1e35","Type":"ContainerStarted","Data":"30e8a4376508f38a133a3efd412f43e1d89e74eb9f8b496b12f559e8e8462d87"}
Jun 15 06:11:05.959722 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:05.959648    2572 generic.go:356] "Generic (PLEG): container finished" podID="5c64dc4b-a424-4464-9e45-253577db1e35" containerID="a7b3c75bbe892905eb59b1793328c316484eea23cdf5485c09055161431ba1da" exitCode=0
Jun 15 06:11:05.960062 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:05.959719    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5" event={"ID":"5c64dc4b-a424-4464-9e45-253577db1e35","Type":"ContainerDied","Data":"a7b3c75bbe892905eb59b1793328c316484eea23cdf5485c09055161431ba1da"}
Jun 15 06:11:06.966790 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:06.966756    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5" event={"ID":"5c64dc4b-a424-4464-9e45-253577db1e35","Type":"ContainerStarted","Data":"706482dbe64918407c77c49a633420fb3338026f56ba9c1b4fc064b3a3408cf0"}
Jun 15 06:11:06.967160 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:06.966818    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5" event={"ID":"5c64dc4b-a424-4464-9e45-253577db1e35","Type":"ContainerStarted","Data":"3f6112a63194e4f00d38d82b67fb0fa4b50e2da88f53c388e02706aa6186474a"}
Jun 15 06:11:06.967160 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:06.967141    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"
Jun 15 06:11:06.967267 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:06.967207    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"
Jun 15 06:11:07.002577 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:07.002521    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5" podStartSLOduration=2.870365139 podStartE2EDuration="7.002505257s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.132140118s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:11:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:11:07.000559232 +0000 UTC m=+1382.148739251" watchObservedRunningTime="2026-06-15 06:11:07.002505257 +0000 UTC m=+1382.150685256"
Jun 15 06:11:12.978037 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:12.978008    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"
Jun 15 06:11:42.981911 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:42.981883    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"
Jun 15 06:11:50.444475 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.444389    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"]
Jun 15 06:11:50.444978 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.444720    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5" podUID="5c64dc4b-a424-4464-9e45-253577db1e35" containerName="kserve-container" containerID="cri-o://3f6112a63194e4f00d38d82b67fb0fa4b50e2da88f53c388e02706aa6186474a" gracePeriod=30
Jun 15 06:11:50.444978 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.444771    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5" podUID="5c64dc4b-a424-4464-9e45-253577db1e35" containerName="kube-rbac-proxy" containerID="cri-o://706482dbe64918407c77c49a633420fb3338026f56ba9c1b4fc064b3a3408cf0" gracePeriod=30
Jun 15 06:11:50.507918 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.507877    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"]
Jun 15 06:11:50.508327 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.508308    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="058a371b-dd05-4591-ba5d-271dfa9244c1" containerName="kserve-container"
Jun 15 06:11:50.508452 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.508329    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="058a371b-dd05-4591-ba5d-271dfa9244c1" containerName="kserve-container"
Jun 15 06:11:50.508452 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.508342    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="058a371b-dd05-4591-ba5d-271dfa9244c1" containerName="kube-rbac-proxy"
Jun 15 06:11:50.508452 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.508349    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="058a371b-dd05-4591-ba5d-271dfa9244c1" containerName="kube-rbac-proxy"
Jun 15 06:11:50.508452 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.508378    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="058a371b-dd05-4591-ba5d-271dfa9244c1" containerName="storage-initializer"
Jun 15 06:11:50.508452 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.508387    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="058a371b-dd05-4591-ba5d-271dfa9244c1" containerName="storage-initializer"
Jun 15 06:11:50.508711 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.508491    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="058a371b-dd05-4591-ba5d-271dfa9244c1" containerName="kserve-container"
Jun 15 06:11:50.508711 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.508510    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="058a371b-dd05-4591-ba5d-271dfa9244c1" containerName="kube-rbac-proxy"
Jun 15 06:11:50.511120 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.511098    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"
Jun 15 06:11:50.513079 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.513059    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-kube-rbac-proxy-sar-config\""
Jun 15 06:11:50.513183 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.513059    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-mcp-predictor-serving-cert\""
Jun 15 06:11:50.524324 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.524297    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"]
Jun 15 06:11:50.641088 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.641050    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab0c04b2-a77e-4719-897b-0b86e7048d3d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-549c7c79f5-r9228\" (UID: \"ab0c04b2-a77e-4719-897b-0b86e7048d3d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"
Jun 15 06:11:50.641290 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.641108    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqctf\" (UniqueName: \"kubernetes.io/projected/ab0c04b2-a77e-4719-897b-0b86e7048d3d-kube-api-access-gqctf\") pod \"isvc-sklearn-mcp-predictor-549c7c79f5-r9228\" (UID: \"ab0c04b2-a77e-4719-897b-0b86e7048d3d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"
Jun 15 06:11:50.641290 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.641145    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab0c04b2-a77e-4719-897b-0b86e7048d3d-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-549c7c79f5-r9228\" (UID: \"ab0c04b2-a77e-4719-897b-0b86e7048d3d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"
Jun 15 06:11:50.641290 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.641178    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab0c04b2-a77e-4719-897b-0b86e7048d3d-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-549c7c79f5-r9228\" (UID: \"ab0c04b2-a77e-4719-897b-0b86e7048d3d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"
Jun 15 06:11:50.641290 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.641214    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab0c04b2-a77e-4719-897b-0b86e7048d3d-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-549c7c79f5-r9228\" (UID: \"ab0c04b2-a77e-4719-897b-0b86e7048d3d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"
Jun 15 06:11:50.742726 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.742628    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab0c04b2-a77e-4719-897b-0b86e7048d3d-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-549c7c79f5-r9228\" (UID: \"ab0c04b2-a77e-4719-897b-0b86e7048d3d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"
Jun 15 06:11:50.742726 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.742704    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab0c04b2-a77e-4719-897b-0b86e7048d3d-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-549c7c79f5-r9228\" (UID: \"ab0c04b2-a77e-4719-897b-0b86e7048d3d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"
Jun 15 06:11:50.743045 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.742823    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab0c04b2-a77e-4719-897b-0b86e7048d3d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-549c7c79f5-r9228\" (UID: \"ab0c04b2-a77e-4719-897b-0b86e7048d3d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"
Jun 15 06:11:50.743045 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.742867    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqctf\" (UniqueName: \"kubernetes.io/projected/ab0c04b2-a77e-4719-897b-0b86e7048d3d-kube-api-access-gqctf\") pod \"isvc-sklearn-mcp-predictor-549c7c79f5-r9228\" (UID: \"ab0c04b2-a77e-4719-897b-0b86e7048d3d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"
Jun 15 06:11:50.743045 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.742908    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab0c04b2-a77e-4719-897b-0b86e7048d3d-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-549c7c79f5-r9228\" (UID: \"ab0c04b2-a77e-4719-897b-0b86e7048d3d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"
Jun 15 06:11:50.743202 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.743147    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab0c04b2-a77e-4719-897b-0b86e7048d3d-kserve-provision-location\") pod \"isvc-sklearn-mcp-predictor-549c7c79f5-r9228\" (UID: \"ab0c04b2-a77e-4719-897b-0b86e7048d3d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"
Jun 15 06:11:50.743722 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.743693    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab0c04b2-a77e-4719-897b-0b86e7048d3d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-mcp-predictor-549c7c79f5-r9228\" (UID: \"ab0c04b2-a77e-4719-897b-0b86e7048d3d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"
Jun 15 06:11:50.743848 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.743734    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab0c04b2-a77e-4719-897b-0b86e7048d3d-cabundle-cert\") pod \"isvc-sklearn-mcp-predictor-549c7c79f5-r9228\" (UID: \"ab0c04b2-a77e-4719-897b-0b86e7048d3d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"
Jun 15 06:11:50.745618 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.745590    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab0c04b2-a77e-4719-897b-0b86e7048d3d-proxy-tls\") pod \"isvc-sklearn-mcp-predictor-549c7c79f5-r9228\" (UID: \"ab0c04b2-a77e-4719-897b-0b86e7048d3d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"
Jun 15 06:11:50.752089 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.752046    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqctf\" (UniqueName: \"kubernetes.io/projected/ab0c04b2-a77e-4719-897b-0b86e7048d3d-kube-api-access-gqctf\") pod \"isvc-sklearn-mcp-predictor-549c7c79f5-r9228\" (UID: \"ab0c04b2-a77e-4719-897b-0b86e7048d3d\") " pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"
Jun 15 06:11:50.821674 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.821635    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"
Jun 15 06:11:50.964215 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:50.964190    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"]
Jun 15 06:11:50.966115 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:11:50.966080    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab0c04b2_a77e_4719_897b_0b86e7048d3d.slice/crio-ee305f9c98bbc06e28652cab4387e0060dd672faffbf42be0041b281662bab81 WatchSource:0}: Error finding container ee305f9c98bbc06e28652cab4387e0060dd672faffbf42be0041b281662bab81: Status 404 returned error can't find the container with id ee305f9c98bbc06e28652cab4387e0060dd672faffbf42be0041b281662bab81
Jun 15 06:11:51.178599 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:51.178564    2572 generic.go:356] "Generic (PLEG): container finished" podID="5c64dc4b-a424-4464-9e45-253577db1e35" containerID="706482dbe64918407c77c49a633420fb3338026f56ba9c1b4fc064b3a3408cf0" exitCode=2
Jun 15 06:11:51.178772 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:51.178637    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5" event={"ID":"5c64dc4b-a424-4464-9e45-253577db1e35","Type":"ContainerDied","Data":"706482dbe64918407c77c49a633420fb3338026f56ba9c1b4fc064b3a3408cf0"}
Jun 15 06:11:51.180383 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:51.180337    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228" event={"ID":"ab0c04b2-a77e-4719-897b-0b86e7048d3d","Type":"ContainerStarted","Data":"ae7254ce214fdaf8f3b22f31a043febd6281574d2c71f39b6f9eda68706399b8"}
Jun 15 06:11:51.180510 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:51.180427    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228" event={"ID":"ab0c04b2-a77e-4719-897b-0b86e7048d3d","Type":"ContainerStarted","Data":"ee305f9c98bbc06e28652cab4387e0060dd672faffbf42be0041b281662bab81"}
Jun 15 06:11:51.685190 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:51.685168    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"
Jun 15 06:11:51.754149 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:51.754116    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c64dc4b-a424-4464-9e45-253577db1e35-cabundle-cert\") pod \"5c64dc4b-a424-4464-9e45-253577db1e35\" (UID: \"5c64dc4b-a424-4464-9e45-253577db1e35\") "
Jun 15 06:11:51.754320 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:51.754161    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c64dc4b-a424-4464-9e45-253577db1e35-kube-rbac-proxy-sar-config\") pod \"5c64dc4b-a424-4464-9e45-253577db1e35\" (UID: \"5c64dc4b-a424-4464-9e45-253577db1e35\") "
Jun 15 06:11:51.754320 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:51.754207    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drgzn\" (UniqueName: \"kubernetes.io/projected/5c64dc4b-a424-4464-9e45-253577db1e35-kube-api-access-drgzn\") pod \"5c64dc4b-a424-4464-9e45-253577db1e35\" (UID: \"5c64dc4b-a424-4464-9e45-253577db1e35\") "
Jun 15 06:11:51.754320 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:51.754242    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c64dc4b-a424-4464-9e45-253577db1e35-proxy-tls\") pod \"5c64dc4b-a424-4464-9e45-253577db1e35\" (UID: \"5c64dc4b-a424-4464-9e45-253577db1e35\") "
Jun 15 06:11:51.754320 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:51.754274    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c64dc4b-a424-4464-9e45-253577db1e35-kserve-provision-location\") pod \"5c64dc4b-a424-4464-9e45-253577db1e35\" (UID: \"5c64dc4b-a424-4464-9e45-253577db1e35\") "
Jun 15 06:11:51.754579 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:51.754551    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c64dc4b-a424-4464-9e45-253577db1e35-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5c64dc4b-a424-4464-9e45-253577db1e35" (UID: "5c64dc4b-a424-4464-9e45-253577db1e35"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:11:51.754646 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:51.754615    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c64dc4b-a424-4464-9e45-253577db1e35-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5c64dc4b-a424-4464-9e45-253577db1e35" (UID: "5c64dc4b-a424-4464-9e45-253577db1e35"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:11:51.754713 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:51.754691    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c64dc4b-a424-4464-9e45-253577db1e35-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5c64dc4b-a424-4464-9e45-253577db1e35" (UID: "5c64dc4b-a424-4464-9e45-253577db1e35"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:11:51.756351 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:51.756327    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c64dc4b-a424-4464-9e45-253577db1e35-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5c64dc4b-a424-4464-9e45-253577db1e35" (UID: "5c64dc4b-a424-4464-9e45-253577db1e35"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:11:51.756473 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:51.756409    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c64dc4b-a424-4464-9e45-253577db1e35-kube-api-access-drgzn" (OuterVolumeSpecName: "kube-api-access-drgzn") pod "5c64dc4b-a424-4464-9e45-253577db1e35" (UID: "5c64dc4b-a424-4464-9e45-253577db1e35"). InnerVolumeSpecName "kube-api-access-drgzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:11:51.855342 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:51.855302    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c64dc4b-a424-4464-9e45-253577db1e35-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:11:51.855342 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:51.855338    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c64dc4b-a424-4464-9e45-253577db1e35-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:11:51.855342 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:51.855348    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drgzn\" (UniqueName: \"kubernetes.io/projected/5c64dc4b-a424-4464-9e45-253577db1e35-kube-api-access-drgzn\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:11:51.855342 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:51.855377    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c64dc4b-a424-4464-9e45-253577db1e35-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:11:51.855627 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:51.855393    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c64dc4b-a424-4464-9e45-253577db1e35-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:11:52.186933 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:52.186902    2572 generic.go:356] "Generic (PLEG): container finished" podID="5c64dc4b-a424-4464-9e45-253577db1e35" containerID="3f6112a63194e4f00d38d82b67fb0fa4b50e2da88f53c388e02706aa6186474a" exitCode=0
Jun 15 06:11:52.187094 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:52.186961    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"
Jun 15 06:11:52.187094 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:52.186986    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5" event={"ID":"5c64dc4b-a424-4464-9e45-253577db1e35","Type":"ContainerDied","Data":"3f6112a63194e4f00d38d82b67fb0fa4b50e2da88f53c388e02706aa6186474a"}
Jun 15 06:11:52.187094 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:52.187055    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5" event={"ID":"5c64dc4b-a424-4464-9e45-253577db1e35","Type":"ContainerDied","Data":"30e8a4376508f38a133a3efd412f43e1d89e74eb9f8b496b12f559e8e8462d87"}
Jun 15 06:11:52.187209 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:52.187093    2572 scope.go:117] "RemoveContainer" containerID="706482dbe64918407c77c49a633420fb3338026f56ba9c1b4fc064b3a3408cf0"
Jun 15 06:11:52.195928 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:52.195908    2572 scope.go:117] "RemoveContainer" containerID="3f6112a63194e4f00d38d82b67fb0fa4b50e2da88f53c388e02706aa6186474a"
Jun 15 06:11:52.204079 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:52.204062    2572 scope.go:117] "RemoveContainer" containerID="a7b3c75bbe892905eb59b1793328c316484eea23cdf5485c09055161431ba1da"
Jun 15 06:11:52.209710 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:52.209684    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"]
Jun 15 06:11:52.212094 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:52.212070    2572 scope.go:117] "RemoveContainer" containerID="706482dbe64918407c77c49a633420fb3338026f56ba9c1b4fc064b3a3408cf0"
Jun 15 06:11:52.212430 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:11:52.212403    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"706482dbe64918407c77c49a633420fb3338026f56ba9c1b4fc064b3a3408cf0\": container with ID starting with 706482dbe64918407c77c49a633420fb3338026f56ba9c1b4fc064b3a3408cf0 not found: ID does not exist" containerID="706482dbe64918407c77c49a633420fb3338026f56ba9c1b4fc064b3a3408cf0"
Jun 15 06:11:52.212517 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:52.212444    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"706482dbe64918407c77c49a633420fb3338026f56ba9c1b4fc064b3a3408cf0"} err="failed to get container status \"706482dbe64918407c77c49a633420fb3338026f56ba9c1b4fc064b3a3408cf0\": rpc error: code = NotFound desc = could not find container \"706482dbe64918407c77c49a633420fb3338026f56ba9c1b4fc064b3a3408cf0\": container with ID starting with 706482dbe64918407c77c49a633420fb3338026f56ba9c1b4fc064b3a3408cf0 not found: ID does not exist"
Jun 15 06:11:52.212517 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:52.212469    2572 scope.go:117] "RemoveContainer" containerID="3f6112a63194e4f00d38d82b67fb0fa4b50e2da88f53c388e02706aa6186474a"
Jun 15 06:11:52.212761 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:11:52.212735    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f6112a63194e4f00d38d82b67fb0fa4b50e2da88f53c388e02706aa6186474a\": container with ID starting with 3f6112a63194e4f00d38d82b67fb0fa4b50e2da88f53c388e02706aa6186474a not found: ID does not exist" containerID="3f6112a63194e4f00d38d82b67fb0fa4b50e2da88f53c388e02706aa6186474a"
Jun 15 06:11:52.212847 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:52.212768    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f6112a63194e4f00d38d82b67fb0fa4b50e2da88f53c388e02706aa6186474a"} err="failed to get container status \"3f6112a63194e4f00d38d82b67fb0fa4b50e2da88f53c388e02706aa6186474a\": rpc error: code = NotFound desc = could not find container \"3f6112a63194e4f00d38d82b67fb0fa4b50e2da88f53c388e02706aa6186474a\": container with ID starting with 3f6112a63194e4f00d38d82b67fb0fa4b50e2da88f53c388e02706aa6186474a not found: ID does not exist"
Jun 15 06:11:52.212847 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:52.212793    2572 scope.go:117] "RemoveContainer" containerID="a7b3c75bbe892905eb59b1793328c316484eea23cdf5485c09055161431ba1da"
Jun 15 06:11:52.213059 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:11:52.213035    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7b3c75bbe892905eb59b1793328c316484eea23cdf5485c09055161431ba1da\": container with ID starting with a7b3c75bbe892905eb59b1793328c316484eea23cdf5485c09055161431ba1da not found: ID does not exist" containerID="a7b3c75bbe892905eb59b1793328c316484eea23cdf5485c09055161431ba1da"
Jun 15 06:11:52.213101 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:52.213070    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7b3c75bbe892905eb59b1793328c316484eea23cdf5485c09055161431ba1da"} err="failed to get container status \"a7b3c75bbe892905eb59b1793328c316484eea23cdf5485c09055161431ba1da\": rpc error: code = NotFound desc = could not find container \"a7b3c75bbe892905eb59b1793328c316484eea23cdf5485c09055161431ba1da\": container with ID starting with a7b3c75bbe892905eb59b1793328c316484eea23cdf5485c09055161431ba1da not found: ID does not exist"
Jun 15 06:11:52.214279 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:52.214260    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-mlflow-v2-runtime-predictor-6456f94494-2p8m5"]
Jun 15 06:11:53.329640 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:53.329607    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c64dc4b-a424-4464-9e45-253577db1e35" path="/var/lib/kubelet/pods/5c64dc4b-a424-4464-9e45-253577db1e35/volumes"
Jun 15 06:11:55.202738 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:55.202707    2572 generic.go:356] "Generic (PLEG): container finished" podID="ab0c04b2-a77e-4719-897b-0b86e7048d3d" containerID="ae7254ce214fdaf8f3b22f31a043febd6281574d2c71f39b6f9eda68706399b8" exitCode=0
Jun 15 06:11:55.203091 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:55.202781    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228" event={"ID":"ab0c04b2-a77e-4719-897b-0b86e7048d3d","Type":"ContainerDied","Data":"ae7254ce214fdaf8f3b22f31a043febd6281574d2c71f39b6f9eda68706399b8"}
Jun 15 06:11:56.212485 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:56.212452    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228" event={"ID":"ab0c04b2-a77e-4719-897b-0b86e7048d3d","Type":"ContainerStarted","Data":"cf4c2247f85817d121f2efff379024be0b3e0235f98864e41c4ef0b5a2c5c71f"}
Jun 15 06:11:58.223936 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:58.223896    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228" event={"ID":"ab0c04b2-a77e-4719-897b-0b86e7048d3d","Type":"ContainerStarted","Data":"8076cba05d130cb9fcb1ea3beef2686c1002b4406571299013b6baa52e2084d4"}
Jun 15 06:11:58.224381 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:58.223973    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228" event={"ID":"ab0c04b2-a77e-4719-897b-0b86e7048d3d","Type":"ContainerStarted","Data":"efdc00c8add49b72292bf3f31762fb4e458d70bfb3f4bb18693619ad36316f46"}
Jun 15 06:11:58.224381 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:58.224095    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"
Jun 15 06:11:58.224381 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:58.224140    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"
Jun 15 06:11:58.246918 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:58.246814    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228" podStartSLOduration=1.807920843 podStartE2EDuration="8.246791831s" totalImagesPullingTime="2.720660356s" totalInitContainerRuntime="3.718210632s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:11:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:11:58.246290647 +0000 UTC m=+1433.394470676" watchObservedRunningTime="2026-06-15 06:11:58.246791831 +0000 UTC m=+1433.394971829"
Jun 15 06:11:59.228105 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:11:59.228070    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"
Jun 15 06:12:05.237977 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:12:05.237943    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"
Jun 15 06:12:25.239254 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:12:25.239210    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228" podUID="ab0c04b2-a77e-4719-897b-0b86e7048d3d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.38:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.38:8080: connect: connection refused"
Jun 15 06:12:35.240354 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:12:35.240322    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"
Jun 15 06:13:05.241908 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:05.241880    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"
Jun 15 06:13:05.390992 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:05.390965    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 06:13:05.397636 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:05.397614    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 06:13:10.601293 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.601259    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"]
Jun 15 06:13:10.601919 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.601557    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228" podUID="ab0c04b2-a77e-4719-897b-0b86e7048d3d" containerName="kserve-container" containerID="cri-o://cf4c2247f85817d121f2efff379024be0b3e0235f98864e41c4ef0b5a2c5c71f" gracePeriod=30
Jun 15 06:13:10.601919 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.601609    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228" podUID="ab0c04b2-a77e-4719-897b-0b86e7048d3d" containerName="kserve-agent" containerID="cri-o://efdc00c8add49b72292bf3f31762fb4e458d70bfb3f4bb18693619ad36316f46" gracePeriod=30
Jun 15 06:13:10.601919 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.601604    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228" podUID="ab0c04b2-a77e-4719-897b-0b86e7048d3d" containerName="kube-rbac-proxy" containerID="cri-o://8076cba05d130cb9fcb1ea3beef2686c1002b4406571299013b6baa52e2084d4" gracePeriod=30
Jun 15 06:13:10.679696 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.679653    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v"]
Jun 15 06:13:10.680109 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.680090    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c64dc4b-a424-4464-9e45-253577db1e35" containerName="kserve-container"
Jun 15 06:13:10.680198 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.680112    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c64dc4b-a424-4464-9e45-253577db1e35" containerName="kserve-container"
Jun 15 06:13:10.680198 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.680141    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c64dc4b-a424-4464-9e45-253577db1e35" containerName="kube-rbac-proxy"
Jun 15 06:13:10.680198 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.680150    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c64dc4b-a424-4464-9e45-253577db1e35" containerName="kube-rbac-proxy"
Jun 15 06:13:10.680198 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.680163    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c64dc4b-a424-4464-9e45-253577db1e35" containerName="storage-initializer"
Jun 15 06:13:10.680198 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.680172    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c64dc4b-a424-4464-9e45-253577db1e35" containerName="storage-initializer"
Jun 15 06:13:10.680512 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.680280    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c64dc4b-a424-4464-9e45-253577db1e35" containerName="kube-rbac-proxy"
Jun 15 06:13:10.680512 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.680295    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c64dc4b-a424-4464-9e45-253577db1e35" containerName="kserve-container"
Jun 15 06:13:10.683782 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.683760    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v"
Jun 15 06:13:10.686497 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.686473    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Jun 15 06:13:10.686497 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.686492    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Jun 15 06:13:10.698564 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.698541    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v"]
Jun 15 06:13:10.767037 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.766995    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0199cf99-cd82-48ca-924c-d6cb31ef6082-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-qh57v\" (UID: \"0199cf99-cd82-48ca-924c-d6cb31ef6082\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v"
Jun 15 06:13:10.767228 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.767056    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz5cj\" (UniqueName: \"kubernetes.io/projected/0199cf99-cd82-48ca-924c-d6cb31ef6082-kube-api-access-pz5cj\") pod \"isvc-paddle-predictor-6d5c9fd4f6-qh57v\" (UID: \"0199cf99-cd82-48ca-924c-d6cb31ef6082\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v"
Jun 15 06:13:10.767228 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.767089    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0199cf99-cd82-48ca-924c-d6cb31ef6082-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-qh57v\" (UID: \"0199cf99-cd82-48ca-924c-d6cb31ef6082\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v"
Jun 15 06:13:10.767228 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.767125    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0199cf99-cd82-48ca-924c-d6cb31ef6082-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-qh57v\" (UID: \"0199cf99-cd82-48ca-924c-d6cb31ef6082\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v"
Jun 15 06:13:10.767228 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.767218    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0199cf99-cd82-48ca-924c-d6cb31ef6082-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-qh57v\" (UID: \"0199cf99-cd82-48ca-924c-d6cb31ef6082\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v"
Jun 15 06:13:10.868274 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.868180    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0199cf99-cd82-48ca-924c-d6cb31ef6082-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-qh57v\" (UID: \"0199cf99-cd82-48ca-924c-d6cb31ef6082\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v"
Jun 15 06:13:10.868466 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.868271    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0199cf99-cd82-48ca-924c-d6cb31ef6082-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-qh57v\" (UID: \"0199cf99-cd82-48ca-924c-d6cb31ef6082\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v"
Jun 15 06:13:10.868466 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.868319    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0199cf99-cd82-48ca-924c-d6cb31ef6082-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-qh57v\" (UID: \"0199cf99-cd82-48ca-924c-d6cb31ef6082\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v"
Jun 15 06:13:10.868466 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.868388    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz5cj\" (UniqueName: \"kubernetes.io/projected/0199cf99-cd82-48ca-924c-d6cb31ef6082-kube-api-access-pz5cj\") pod \"isvc-paddle-predictor-6d5c9fd4f6-qh57v\" (UID: \"0199cf99-cd82-48ca-924c-d6cb31ef6082\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v"
Jun 15 06:13:10.868644 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.868480    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0199cf99-cd82-48ca-924c-d6cb31ef6082-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-qh57v\" (UID: \"0199cf99-cd82-48ca-924c-d6cb31ef6082\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v"
Jun 15 06:13:10.868767 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.868748    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0199cf99-cd82-48ca-924c-d6cb31ef6082-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-qh57v\" (UID: \"0199cf99-cd82-48ca-924c-d6cb31ef6082\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v"
Jun 15 06:13:10.868983 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.868953    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0199cf99-cd82-48ca-924c-d6cb31ef6082-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-qh57v\" (UID: \"0199cf99-cd82-48ca-924c-d6cb31ef6082\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v"
Jun 15 06:13:10.869113 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.869094    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0199cf99-cd82-48ca-924c-d6cb31ef6082-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-qh57v\" (UID: \"0199cf99-cd82-48ca-924c-d6cb31ef6082\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v"
Jun 15 06:13:10.870968 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.870947    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0199cf99-cd82-48ca-924c-d6cb31ef6082-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-qh57v\" (UID: \"0199cf99-cd82-48ca-924c-d6cb31ef6082\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v"
Jun 15 06:13:10.877545 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.877523    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz5cj\" (UniqueName: \"kubernetes.io/projected/0199cf99-cd82-48ca-924c-d6cb31ef6082-kube-api-access-pz5cj\") pod \"isvc-paddle-predictor-6d5c9fd4f6-qh57v\" (UID: \"0199cf99-cd82-48ca-924c-d6cb31ef6082\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v"
Jun 15 06:13:10.994539 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:10.994506    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v"
Jun 15 06:13:11.128925 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:11.128901    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v"]
Jun 15 06:13:11.131091 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:13:11.131064    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0199cf99_cd82_48ca_924c_d6cb31ef6082.slice/crio-499dc69223f6039ba393d583889cc2051d41bf16b81a71fc226219669337296f WatchSource:0}: Error finding container 499dc69223f6039ba393d583889cc2051d41bf16b81a71fc226219669337296f: Status 404 returned error can't find the container with id 499dc69223f6039ba393d583889cc2051d41bf16b81a71fc226219669337296f
Jun 15 06:13:11.553736 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:11.553702    2572 generic.go:356] "Generic (PLEG): container finished" podID="ab0c04b2-a77e-4719-897b-0b86e7048d3d" containerID="8076cba05d130cb9fcb1ea3beef2686c1002b4406571299013b6baa52e2084d4" exitCode=2
Jun 15 06:13:11.553935 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:11.553782    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228" event={"ID":"ab0c04b2-a77e-4719-897b-0b86e7048d3d","Type":"ContainerDied","Data":"8076cba05d130cb9fcb1ea3beef2686c1002b4406571299013b6baa52e2084d4"}
Jun 15 06:13:11.555376 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:11.555335    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v" event={"ID":"0199cf99-cd82-48ca-924c-d6cb31ef6082","Type":"ContainerStarted","Data":"c646b0ee7af15b01d86950cda2ecca5bbd0259bace0ef13ae9072ebf4f759628"}
Jun 15 06:13:11.555512 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:11.555413    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v" event={"ID":"0199cf99-cd82-48ca-924c-d6cb31ef6082","Type":"ContainerStarted","Data":"499dc69223f6039ba393d583889cc2051d41bf16b81a71fc226219669337296f"}
Jun 15 06:13:13.566561 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:13.566525    2572 generic.go:356] "Generic (PLEG): container finished" podID="ab0c04b2-a77e-4719-897b-0b86e7048d3d" containerID="cf4c2247f85817d121f2efff379024be0b3e0235f98864e41c4ef0b5a2c5c71f" exitCode=0
Jun 15 06:13:13.566899 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:13.566595    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228" event={"ID":"ab0c04b2-a77e-4719-897b-0b86e7048d3d","Type":"ContainerDied","Data":"cf4c2247f85817d121f2efff379024be0b3e0235f98864e41c4ef0b5a2c5c71f"}
Jun 15 06:13:15.233512 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:15.233427    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228" podUID="ab0c04b2-a77e-4719-897b-0b86e7048d3d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jun 15 06:13:15.238865 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:15.238833    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228" podUID="ab0c04b2-a77e-4719-897b-0b86e7048d3d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.38:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.38:8080: connect: connection refused"
Jun 15 06:13:16.581533 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:16.581500    2572 generic.go:356] "Generic (PLEG): container finished" podID="0199cf99-cd82-48ca-924c-d6cb31ef6082" containerID="c646b0ee7af15b01d86950cda2ecca5bbd0259bace0ef13ae9072ebf4f759628" exitCode=0
Jun 15 06:13:16.581895 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:16.581570    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v" event={"ID":"0199cf99-cd82-48ca-924c-d6cb31ef6082","Type":"ContainerDied","Data":"c646b0ee7af15b01d86950cda2ecca5bbd0259bace0ef13ae9072ebf4f759628"}
Jun 15 06:13:16.582995 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:16.582977    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 15 06:13:20.233505 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:20.233447    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228" podUID="ab0c04b2-a77e-4719-897b-0b86e7048d3d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jun 15 06:13:25.232656 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:25.232601    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228" podUID="ab0c04b2-a77e-4719-897b-0b86e7048d3d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jun 15 06:13:25.233152 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:25.232735    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"
Jun 15 06:13:25.238736 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:25.238691    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228" podUID="ab0c04b2-a77e-4719-897b-0b86e7048d3d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.38:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.38:8080: connect: connection refused"
Jun 15 06:13:30.233046 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:30.233005    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228" podUID="ab0c04b2-a77e-4719-897b-0b86e7048d3d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jun 15 06:13:30.661418 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:30.661386    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v" event={"ID":"0199cf99-cd82-48ca-924c-d6cb31ef6082","Type":"ContainerStarted","Data":"22441559ee3b3d224ec36c7792ea4675391e12a08935eb478773239eecfd49ef"}
Jun 15 06:13:30.661574 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:30.661446    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v" event={"ID":"0199cf99-cd82-48ca-924c-d6cb31ef6082","Type":"ContainerStarted","Data":"006599045c8841c091e11a34c1b047796110b7acee03768e25b146aebbbbd093"}
Jun 15 06:13:30.661746 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:30.661729    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v"
Jun 15 06:13:30.661923 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:30.661898    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v"
Jun 15 06:13:30.663022 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:30.662998    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v" podUID="0199cf99-cd82-48ca-924c-d6cb31ef6082" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jun 15 06:13:30.683852 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:30.683809    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v" podStartSLOduration=1.691464625 podStartE2EDuration="20.683797195s" totalImagesPullingTime="13.791510256s" totalInitContainerRuntime="5.200822314s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:13:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:13:30.682790974 +0000 UTC m=+1525.830970972" watchObservedRunningTime="2026-06-15 06:13:30.683797195 +0000 UTC m=+1525.831977191"
Jun 15 06:13:31.666331 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:31.666296    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v" podUID="0199cf99-cd82-48ca-924c-d6cb31ef6082" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jun 15 06:13:35.232923 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:35.232882    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228" podUID="ab0c04b2-a77e-4719-897b-0b86e7048d3d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jun 15 06:13:35.239380 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:35.239343    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228" podUID="ab0c04b2-a77e-4719-897b-0b86e7048d3d" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.38:8080/v1/models/isvc-sklearn-mcp\": dial tcp 10.134.0.38:8080: connect: connection refused"
Jun 15 06:13:35.239490 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:35.239452    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"
Jun 15 06:13:36.672269 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:36.672239    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v"
Jun 15 06:13:36.672730 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:36.672705    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v" podUID="0199cf99-cd82-48ca-924c-d6cb31ef6082" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jun 15 06:13:40.233543 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:40.233498    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228" podUID="ab0c04b2-a77e-4719-897b-0b86e7048d3d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.38:8643/healthz\": dial tcp 10.134.0.38:8643: connect: connection refused"
Jun 15 06:13:40.713304 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:40.713275    2572 generic.go:356] "Generic (PLEG): container finished" podID="ab0c04b2-a77e-4719-897b-0b86e7048d3d" containerID="efdc00c8add49b72292bf3f31762fb4e458d70bfb3f4bb18693619ad36316f46" exitCode=0
Jun 15 06:13:40.713475 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:40.713344    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228" event={"ID":"ab0c04b2-a77e-4719-897b-0b86e7048d3d","Type":"ContainerDied","Data":"efdc00c8add49b72292bf3f31762fb4e458d70bfb3f4bb18693619ad36316f46"}
Jun 15 06:13:40.797973 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:40.797950    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"
Jun 15 06:13:40.959983 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:40.959950    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab0c04b2-a77e-4719-897b-0b86e7048d3d-kserve-provision-location\") pod \"ab0c04b2-a77e-4719-897b-0b86e7048d3d\" (UID: \"ab0c04b2-a77e-4719-897b-0b86e7048d3d\") "
Jun 15 06:13:40.960131 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:40.960029    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab0c04b2-a77e-4719-897b-0b86e7048d3d-cabundle-cert\") pod \"ab0c04b2-a77e-4719-897b-0b86e7048d3d\" (UID: \"ab0c04b2-a77e-4719-897b-0b86e7048d3d\") "
Jun 15 06:13:40.960131 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:40.960084    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqctf\" (UniqueName: \"kubernetes.io/projected/ab0c04b2-a77e-4719-897b-0b86e7048d3d-kube-api-access-gqctf\") pod \"ab0c04b2-a77e-4719-897b-0b86e7048d3d\" (UID: \"ab0c04b2-a77e-4719-897b-0b86e7048d3d\") "
Jun 15 06:13:40.960233 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:40.960132    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab0c04b2-a77e-4719-897b-0b86e7048d3d-proxy-tls\") pod \"ab0c04b2-a77e-4719-897b-0b86e7048d3d\" (UID: \"ab0c04b2-a77e-4719-897b-0b86e7048d3d\") "
Jun 15 06:13:40.960233 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:40.960163    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab0c04b2-a77e-4719-897b-0b86e7048d3d-kube-rbac-proxy-sar-config\") pod \"ab0c04b2-a77e-4719-897b-0b86e7048d3d\" (UID: \"ab0c04b2-a77e-4719-897b-0b86e7048d3d\") "
Jun 15 06:13:40.960333 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:40.960314    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab0c04b2-a77e-4719-897b-0b86e7048d3d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ab0c04b2-a77e-4719-897b-0b86e7048d3d" (UID: "ab0c04b2-a77e-4719-897b-0b86e7048d3d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:13:40.960463 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:40.960447    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ab0c04b2-a77e-4719-897b-0b86e7048d3d-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:13:40.960556 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:40.960438    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab0c04b2-a77e-4719-897b-0b86e7048d3d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ab0c04b2-a77e-4719-897b-0b86e7048d3d" (UID: "ab0c04b2-a77e-4719-897b-0b86e7048d3d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:13:40.960639 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:40.960614    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab0c04b2-a77e-4719-897b-0b86e7048d3d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ab0c04b2-a77e-4719-897b-0b86e7048d3d" (UID: "ab0c04b2-a77e-4719-897b-0b86e7048d3d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:13:40.962195 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:40.962172    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab0c04b2-a77e-4719-897b-0b86e7048d3d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ab0c04b2-a77e-4719-897b-0b86e7048d3d" (UID: "ab0c04b2-a77e-4719-897b-0b86e7048d3d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:13:40.962277 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:40.962193    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab0c04b2-a77e-4719-897b-0b86e7048d3d-kube-api-access-gqctf" (OuterVolumeSpecName: "kube-api-access-gqctf") pod "ab0c04b2-a77e-4719-897b-0b86e7048d3d" (UID: "ab0c04b2-a77e-4719-897b-0b86e7048d3d"). InnerVolumeSpecName "kube-api-access-gqctf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:13:41.061571 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:41.061534    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ab0c04b2-a77e-4719-897b-0b86e7048d3d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:13:41.061571 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:41.061564    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ab0c04b2-a77e-4719-897b-0b86e7048d3d-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:13:41.061571 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:41.061575    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqctf\" (UniqueName: \"kubernetes.io/projected/ab0c04b2-a77e-4719-897b-0b86e7048d3d-kube-api-access-gqctf\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:13:41.061800 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:41.061583    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ab0c04b2-a77e-4719-897b-0b86e7048d3d-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:13:41.720548 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:41.720520    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228" event={"ID":"ab0c04b2-a77e-4719-897b-0b86e7048d3d","Type":"ContainerDied","Data":"ee305f9c98bbc06e28652cab4387e0060dd672faffbf42be0041b281662bab81"}
Jun 15 06:13:41.720548 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:41.720553    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"
Jun 15 06:13:41.721108 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:41.720595    2572 scope.go:117] "RemoveContainer" containerID="8076cba05d130cb9fcb1ea3beef2686c1002b4406571299013b6baa52e2084d4"
Jun 15 06:13:41.729027 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:41.729010    2572 scope.go:117] "RemoveContainer" containerID="efdc00c8add49b72292bf3f31762fb4e458d70bfb3f4bb18693619ad36316f46"
Jun 15 06:13:41.736372 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:41.736335    2572 scope.go:117] "RemoveContainer" containerID="cf4c2247f85817d121f2efff379024be0b3e0235f98864e41c4ef0b5a2c5c71f"
Jun 15 06:13:41.744227 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:41.744209    2572 scope.go:117] "RemoveContainer" containerID="ae7254ce214fdaf8f3b22f31a043febd6281574d2c71f39b6f9eda68706399b8"
Jun 15 06:13:41.745483 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:41.745458    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"]
Jun 15 06:13:41.752998 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:41.752982    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-mcp-predictor-549c7c79f5-r9228"]
Jun 15 06:13:43.324214 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:43.324178    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab0c04b2-a77e-4719-897b-0b86e7048d3d" path="/var/lib/kubelet/pods/ab0c04b2-a77e-4719-897b-0b86e7048d3d/volumes"
Jun 15 06:13:46.672770 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:46.672733    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v" podUID="0199cf99-cd82-48ca-924c-d6cb31ef6082" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jun 15 06:13:56.672825 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:13:56.672787    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v" podUID="0199cf99-cd82-48ca-924c-d6cb31ef6082" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jun 15 06:14:06.673388 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:06.673336    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v" podUID="0199cf99-cd82-48ca-924c-d6cb31ef6082" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jun 15 06:14:16.674079 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:16.674044    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v"
Jun 15 06:14:21.793093 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:21.793058    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v"]
Jun 15 06:14:21.793586 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:21.793280    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v" podUID="0199cf99-cd82-48ca-924c-d6cb31ef6082" containerName="kserve-container" containerID="cri-o://006599045c8841c091e11a34c1b047796110b7acee03768e25b146aebbbbd093" gracePeriod=30
Jun 15 06:14:21.793586 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:21.793329    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v" podUID="0199cf99-cd82-48ca-924c-d6cb31ef6082" containerName="kube-rbac-proxy" containerID="cri-o://22441559ee3b3d224ec36c7792ea4675391e12a08935eb478773239eecfd49ef" gracePeriod=30
Jun 15 06:14:21.884204 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:21.884161    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h"]
Jun 15 06:14:21.884629 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:21.884610    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab0c04b2-a77e-4719-897b-0b86e7048d3d" containerName="kserve-container"
Jun 15 06:14:21.884720 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:21.884630    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab0c04b2-a77e-4719-897b-0b86e7048d3d" containerName="kserve-container"
Jun 15 06:14:21.884720 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:21.884669    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab0c04b2-a77e-4719-897b-0b86e7048d3d" containerName="kserve-agent"
Jun 15 06:14:21.884720 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:21.884679    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab0c04b2-a77e-4719-897b-0b86e7048d3d" containerName="kserve-agent"
Jun 15 06:14:21.884720 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:21.884697    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab0c04b2-a77e-4719-897b-0b86e7048d3d" containerName="storage-initializer"
Jun 15 06:14:21.884720 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:21.884706    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab0c04b2-a77e-4719-897b-0b86e7048d3d" containerName="storage-initializer"
Jun 15 06:14:21.884720 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:21.884716    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab0c04b2-a77e-4719-897b-0b86e7048d3d" containerName="kube-rbac-proxy"
Jun 15 06:14:21.885021 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:21.884725    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab0c04b2-a77e-4719-897b-0b86e7048d3d" containerName="kube-rbac-proxy"
Jun 15 06:14:21.885021 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:21.884809    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab0c04b2-a77e-4719-897b-0b86e7048d3d" containerName="kserve-agent"
Jun 15 06:14:21.885021 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:21.884823    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab0c04b2-a77e-4719-897b-0b86e7048d3d" containerName="kube-rbac-proxy"
Jun 15 06:14:21.885021 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:21.884836    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab0c04b2-a77e-4719-897b-0b86e7048d3d" containerName="kserve-container"
Jun 15 06:14:21.908340 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:21.908303    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h"]
Jun 15 06:14:21.908531 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:21.908448    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h"
Jun 15 06:14:21.910772 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:21.910748    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Jun 15 06:14:21.910912 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:21.910777    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Jun 15 06:14:21.915507 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:21.915479    2572 generic.go:356] "Generic (PLEG): container finished" podID="0199cf99-cd82-48ca-924c-d6cb31ef6082" containerID="22441559ee3b3d224ec36c7792ea4675391e12a08935eb478773239eecfd49ef" exitCode=2
Jun 15 06:14:21.915650 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:21.915558    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v" event={"ID":"0199cf99-cd82-48ca-924c-d6cb31ef6082","Type":"ContainerDied","Data":"22441559ee3b3d224ec36c7792ea4675391e12a08935eb478773239eecfd49ef"}
Jun 15 06:14:22.033796 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:22.033746    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f123126-bb80-4b22-bc54-abd0cfad587f-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jrz5h\" (UID: \"9f123126-bb80-4b22-bc54-abd0cfad587f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h"
Jun 15 06:14:22.033796 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:22.033803    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f123126-bb80-4b22-bc54-abd0cfad587f-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jrz5h\" (UID: \"9f123126-bb80-4b22-bc54-abd0cfad587f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h"
Jun 15 06:14:22.034087 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:22.033878    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f123126-bb80-4b22-bc54-abd0cfad587f-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jrz5h\" (UID: \"9f123126-bb80-4b22-bc54-abd0cfad587f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h"
Jun 15 06:14:22.034087 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:22.033939    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2bn2\" (UniqueName: \"kubernetes.io/projected/9f123126-bb80-4b22-bc54-abd0cfad587f-kube-api-access-x2bn2\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jrz5h\" (UID: \"9f123126-bb80-4b22-bc54-abd0cfad587f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h"
Jun 15 06:14:22.034087 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:22.033970    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f123126-bb80-4b22-bc54-abd0cfad587f-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jrz5h\" (UID: \"9f123126-bb80-4b22-bc54-abd0cfad587f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h"
Jun 15 06:14:22.134832 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:22.134744    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f123126-bb80-4b22-bc54-abd0cfad587f-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jrz5h\" (UID: \"9f123126-bb80-4b22-bc54-abd0cfad587f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h"
Jun 15 06:14:22.134832 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:22.134795    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f123126-bb80-4b22-bc54-abd0cfad587f-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jrz5h\" (UID: \"9f123126-bb80-4b22-bc54-abd0cfad587f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h"
Jun 15 06:14:22.134832 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:22.134822    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2bn2\" (UniqueName: \"kubernetes.io/projected/9f123126-bb80-4b22-bc54-abd0cfad587f-kube-api-access-x2bn2\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jrz5h\" (UID: \"9f123126-bb80-4b22-bc54-abd0cfad587f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h"
Jun 15 06:14:22.135173 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:22.134838    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f123126-bb80-4b22-bc54-abd0cfad587f-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jrz5h\" (UID: \"9f123126-bb80-4b22-bc54-abd0cfad587f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h"
Jun 15 06:14:22.135173 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:22.134926    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f123126-bb80-4b22-bc54-abd0cfad587f-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jrz5h\" (UID: \"9f123126-bb80-4b22-bc54-abd0cfad587f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h"
Jun 15 06:14:22.135335 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:22.135314    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f123126-bb80-4b22-bc54-abd0cfad587f-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jrz5h\" (UID: \"9f123126-bb80-4b22-bc54-abd0cfad587f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h"
Jun 15 06:14:22.135642 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:22.135614    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f123126-bb80-4b22-bc54-abd0cfad587f-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jrz5h\" (UID: \"9f123126-bb80-4b22-bc54-abd0cfad587f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h"
Jun 15 06:14:22.135748 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:22.135617    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f123126-bb80-4b22-bc54-abd0cfad587f-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jrz5h\" (UID: \"9f123126-bb80-4b22-bc54-abd0cfad587f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h"
Jun 15 06:14:22.137308 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:22.137282    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f123126-bb80-4b22-bc54-abd0cfad587f-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jrz5h\" (UID: \"9f123126-bb80-4b22-bc54-abd0cfad587f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h"
Jun 15 06:14:22.145042 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:22.145020    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2bn2\" (UniqueName: \"kubernetes.io/projected/9f123126-bb80-4b22-bc54-abd0cfad587f-kube-api-access-x2bn2\") pod \"isvc-paddle-runtime-predictor-58b99fff76-jrz5h\" (UID: \"9f123126-bb80-4b22-bc54-abd0cfad587f\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h"
Jun 15 06:14:22.220769 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:22.220731    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h"
Jun 15 06:14:22.354908 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:22.354881    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h"]
Jun 15 06:14:22.358045 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:14:22.358016    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f123126_bb80_4b22_bc54_abd0cfad587f.slice/crio-2dd3e10df63afcd0e70b36a334e7bce973c0c9eddcb1a12e0165931a07a882cc WatchSource:0}: Error finding container 2dd3e10df63afcd0e70b36a334e7bce973c0c9eddcb1a12e0165931a07a882cc: Status 404 returned error can't find the container with id 2dd3e10df63afcd0e70b36a334e7bce973c0c9eddcb1a12e0165931a07a882cc
Jun 15 06:14:22.922799 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:22.922767    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h" event={"ID":"9f123126-bb80-4b22-bc54-abd0cfad587f","Type":"ContainerStarted","Data":"b7d9283a3669e7c26bbb39b9334fee31c8544622fe2a5c06792d062bcd651700"}
Jun 15 06:14:22.923259 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:22.922829    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h" event={"ID":"9f123126-bb80-4b22-bc54-abd0cfad587f","Type":"ContainerStarted","Data":"2dd3e10df63afcd0e70b36a334e7bce973c0c9eddcb1a12e0165931a07a882cc"}
Jun 15 06:14:24.532767 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.532739    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v"
Jun 15 06:14:24.557980 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.557949    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0199cf99-cd82-48ca-924c-d6cb31ef6082-proxy-tls\") pod \"0199cf99-cd82-48ca-924c-d6cb31ef6082\" (UID: \"0199cf99-cd82-48ca-924c-d6cb31ef6082\") "
Jun 15 06:14:24.558122 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.557992    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pz5cj\" (UniqueName: \"kubernetes.io/projected/0199cf99-cd82-48ca-924c-d6cb31ef6082-kube-api-access-pz5cj\") pod \"0199cf99-cd82-48ca-924c-d6cb31ef6082\" (UID: \"0199cf99-cd82-48ca-924c-d6cb31ef6082\") "
Jun 15 06:14:24.558122 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.558019    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0199cf99-cd82-48ca-924c-d6cb31ef6082-cabundle-cert\") pod \"0199cf99-cd82-48ca-924c-d6cb31ef6082\" (UID: \"0199cf99-cd82-48ca-924c-d6cb31ef6082\") "
Jun 15 06:14:24.558122 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.558099    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0199cf99-cd82-48ca-924c-d6cb31ef6082-kube-rbac-proxy-sar-config\") pod \"0199cf99-cd82-48ca-924c-d6cb31ef6082\" (UID: \"0199cf99-cd82-48ca-924c-d6cb31ef6082\") "
Jun 15 06:14:24.558317 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.558146    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0199cf99-cd82-48ca-924c-d6cb31ef6082-kserve-provision-location\") pod \"0199cf99-cd82-48ca-924c-d6cb31ef6082\" (UID: \"0199cf99-cd82-48ca-924c-d6cb31ef6082\") "
Jun 15 06:14:24.558526 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.558421    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0199cf99-cd82-48ca-924c-d6cb31ef6082-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0199cf99-cd82-48ca-924c-d6cb31ef6082" (UID: "0199cf99-cd82-48ca-924c-d6cb31ef6082"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:14:24.558645 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.558598    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0199cf99-cd82-48ca-924c-d6cb31ef6082-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0199cf99-cd82-48ca-924c-d6cb31ef6082" (UID: "0199cf99-cd82-48ca-924c-d6cb31ef6082"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:14:24.560319 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.560290    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0199cf99-cd82-48ca-924c-d6cb31ef6082-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0199cf99-cd82-48ca-924c-d6cb31ef6082" (UID: "0199cf99-cd82-48ca-924c-d6cb31ef6082"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:14:24.560459 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.560394    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0199cf99-cd82-48ca-924c-d6cb31ef6082-kube-api-access-pz5cj" (OuterVolumeSpecName: "kube-api-access-pz5cj") pod "0199cf99-cd82-48ca-924c-d6cb31ef6082" (UID: "0199cf99-cd82-48ca-924c-d6cb31ef6082"). InnerVolumeSpecName "kube-api-access-pz5cj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:14:24.573908 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.573879    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0199cf99-cd82-48ca-924c-d6cb31ef6082-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0199cf99-cd82-48ca-924c-d6cb31ef6082" (UID: "0199cf99-cd82-48ca-924c-d6cb31ef6082"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:14:24.658991 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.658964    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz5cj\" (UniqueName: \"kubernetes.io/projected/0199cf99-cd82-48ca-924c-d6cb31ef6082-kube-api-access-pz5cj\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:14:24.658991 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.658994    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0199cf99-cd82-48ca-924c-d6cb31ef6082-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:14:24.659188 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.659006    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0199cf99-cd82-48ca-924c-d6cb31ef6082-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:14:24.659188 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.659015    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0199cf99-cd82-48ca-924c-d6cb31ef6082-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:14:24.659188 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.659024    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0199cf99-cd82-48ca-924c-d6cb31ef6082-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:14:24.934510 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.934425    2572 generic.go:356] "Generic (PLEG): container finished" podID="0199cf99-cd82-48ca-924c-d6cb31ef6082" containerID="006599045c8841c091e11a34c1b047796110b7acee03768e25b146aebbbbd093" exitCode=0
Jun 15 06:14:24.934510 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.934462    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v" event={"ID":"0199cf99-cd82-48ca-924c-d6cb31ef6082","Type":"ContainerDied","Data":"006599045c8841c091e11a34c1b047796110b7acee03768e25b146aebbbbd093"}
Jun 15 06:14:24.934510 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.934485    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v"
Jun 15 06:14:24.934510 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.934506    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v" event={"ID":"0199cf99-cd82-48ca-924c-d6cb31ef6082","Type":"ContainerDied","Data":"499dc69223f6039ba393d583889cc2051d41bf16b81a71fc226219669337296f"}
Jun 15 06:14:24.934818 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.934532    2572 scope.go:117] "RemoveContainer" containerID="22441559ee3b3d224ec36c7792ea4675391e12a08935eb478773239eecfd49ef"
Jun 15 06:14:24.943263 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.943247    2572 scope.go:117] "RemoveContainer" containerID="006599045c8841c091e11a34c1b047796110b7acee03768e25b146aebbbbd093"
Jun 15 06:14:24.950750 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.950731    2572 scope.go:117] "RemoveContainer" containerID="c646b0ee7af15b01d86950cda2ecca5bbd0259bace0ef13ae9072ebf4f759628"
Jun 15 06:14:24.957966 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.957944    2572 scope.go:117] "RemoveContainer" containerID="22441559ee3b3d224ec36c7792ea4675391e12a08935eb478773239eecfd49ef"
Jun 15 06:14:24.958206 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:14:24.958189    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22441559ee3b3d224ec36c7792ea4675391e12a08935eb478773239eecfd49ef\": container with ID starting with 22441559ee3b3d224ec36c7792ea4675391e12a08935eb478773239eecfd49ef not found: ID does not exist" containerID="22441559ee3b3d224ec36c7792ea4675391e12a08935eb478773239eecfd49ef"
Jun 15 06:14:24.958258 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.958215    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22441559ee3b3d224ec36c7792ea4675391e12a08935eb478773239eecfd49ef"} err="failed to get container status \"22441559ee3b3d224ec36c7792ea4675391e12a08935eb478773239eecfd49ef\": rpc error: code = NotFound desc = could not find container \"22441559ee3b3d224ec36c7792ea4675391e12a08935eb478773239eecfd49ef\": container with ID starting with 22441559ee3b3d224ec36c7792ea4675391e12a08935eb478773239eecfd49ef not found: ID does not exist"
Jun 15 06:14:24.958258 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.958232    2572 scope.go:117] "RemoveContainer" containerID="006599045c8841c091e11a34c1b047796110b7acee03768e25b146aebbbbd093"
Jun 15 06:14:24.958484 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:14:24.958452    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"006599045c8841c091e11a34c1b047796110b7acee03768e25b146aebbbbd093\": container with ID starting with 006599045c8841c091e11a34c1b047796110b7acee03768e25b146aebbbbd093 not found: ID does not exist" containerID="006599045c8841c091e11a34c1b047796110b7acee03768e25b146aebbbbd093"
Jun 15 06:14:24.958582 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.958500    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"006599045c8841c091e11a34c1b047796110b7acee03768e25b146aebbbbd093"} err="failed to get container status \"006599045c8841c091e11a34c1b047796110b7acee03768e25b146aebbbbd093\": rpc error: code = NotFound desc = could not find container \"006599045c8841c091e11a34c1b047796110b7acee03768e25b146aebbbbd093\": container with ID starting with 006599045c8841c091e11a34c1b047796110b7acee03768e25b146aebbbbd093 not found: ID does not exist"
Jun 15 06:14:24.958582 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.958522    2572 scope.go:117] "RemoveContainer" containerID="c646b0ee7af15b01d86950cda2ecca5bbd0259bace0ef13ae9072ebf4f759628"
Jun 15 06:14:24.959187 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:14:24.959162    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c646b0ee7af15b01d86950cda2ecca5bbd0259bace0ef13ae9072ebf4f759628\": container with ID starting with c646b0ee7af15b01d86950cda2ecca5bbd0259bace0ef13ae9072ebf4f759628 not found: ID does not exist" containerID="c646b0ee7af15b01d86950cda2ecca5bbd0259bace0ef13ae9072ebf4f759628"
Jun 15 06:14:24.959285 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.959193    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c646b0ee7af15b01d86950cda2ecca5bbd0259bace0ef13ae9072ebf4f759628"} err="failed to get container status \"c646b0ee7af15b01d86950cda2ecca5bbd0259bace0ef13ae9072ebf4f759628\": rpc error: code = NotFound desc = could not find container \"c646b0ee7af15b01d86950cda2ecca5bbd0259bace0ef13ae9072ebf4f759628\": container with ID starting with c646b0ee7af15b01d86950cda2ecca5bbd0259bace0ef13ae9072ebf4f759628 not found: ID does not exist"
Jun 15 06:14:24.961216 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.961194    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v"]
Jun 15 06:14:24.963680 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:24.963657    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-qh57v"]
Jun 15 06:14:25.323542 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:25.323466    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0199cf99-cd82-48ca-924c-d6cb31ef6082" path="/var/lib/kubelet/pods/0199cf99-cd82-48ca-924c-d6cb31ef6082/volumes"
Jun 15 06:14:26.952537 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:26.952504    2572 generic.go:356] "Generic (PLEG): container finished" podID="9f123126-bb80-4b22-bc54-abd0cfad587f" containerID="b7d9283a3669e7c26bbb39b9334fee31c8544622fe2a5c06792d062bcd651700" exitCode=0
Jun 15 06:14:26.952913 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:26.952585    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h" event={"ID":"9f123126-bb80-4b22-bc54-abd0cfad587f","Type":"ContainerDied","Data":"b7d9283a3669e7c26bbb39b9334fee31c8544622fe2a5c06792d062bcd651700"}
Jun 15 06:14:27.960116 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:27.960077    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h" event={"ID":"9f123126-bb80-4b22-bc54-abd0cfad587f","Type":"ContainerStarted","Data":"ea2f20d0109d2dc4795dec884bbb7aa257424fa09eeac0c1f82b73a8908fd853"}
Jun 15 06:14:27.960518 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:27.960138    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h" event={"ID":"9f123126-bb80-4b22-bc54-abd0cfad587f","Type":"ContainerStarted","Data":"f29f9c10bb09242d3f29e84b6e1227496cbeea4dec2638816536d14aec1c00a1"}
Jun 15 06:14:27.960518 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:27.960389    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h"
Jun 15 06:14:27.983977 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:27.983925    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h" podStartSLOduration=2.91010379 podStartE2EDuration="6.983909358s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.073805568s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:14:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:14:27.983147218 +0000 UTC m=+1583.131327226" watchObservedRunningTime="2026-06-15 06:14:27.983909358 +0000 UTC m=+1583.132089414"
Jun 15 06:14:28.964988 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:28.964957    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h"
Jun 15 06:14:28.966235 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:28.966209    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h" podUID="9f123126-bb80-4b22-bc54-abd0cfad587f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jun 15 06:14:29.969880 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:29.969838    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h" podUID="9f123126-bb80-4b22-bc54-abd0cfad587f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jun 15 06:14:34.974007 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:34.973980    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h"
Jun 15 06:14:34.974518 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:34.974497    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h" podUID="9f123126-bb80-4b22-bc54-abd0cfad587f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jun 15 06:14:44.975135 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:44.975097    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h" podUID="9f123126-bb80-4b22-bc54-abd0cfad587f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jun 15 06:14:54.975070 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:14:54.975028    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h" podUID="9f123126-bb80-4b22-bc54-abd0cfad587f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jun 15 06:15:04.974456 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:04.974414    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h" podUID="9f123126-bb80-4b22-bc54-abd0cfad587f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jun 15 06:15:14.975088 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:14.975052    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h"
Jun 15 06:15:22.998103 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:22.998065    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h"]
Jun 15 06:15:22.998556 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:22.998330    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h" podUID="9f123126-bb80-4b22-bc54-abd0cfad587f" containerName="kserve-container" containerID="cri-o://f29f9c10bb09242d3f29e84b6e1227496cbeea4dec2638816536d14aec1c00a1" gracePeriod=30
Jun 15 06:15:22.998556 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:22.998346    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h" podUID="9f123126-bb80-4b22-bc54-abd0cfad587f" containerName="kube-rbac-proxy" containerID="cri-o://ea2f20d0109d2dc4795dec884bbb7aa257424fa09eeac0c1f82b73a8908fd853" gracePeriod=30
Jun 15 06:15:23.082512 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.082472    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"]
Jun 15 06:15:23.082861 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.082839    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0199cf99-cd82-48ca-924c-d6cb31ef6082" containerName="kserve-container"
Jun 15 06:15:23.082861 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.082854    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0199cf99-cd82-48ca-924c-d6cb31ef6082" containerName="kserve-container"
Jun 15 06:15:23.082861 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.082862    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0199cf99-cd82-48ca-924c-d6cb31ef6082" containerName="storage-initializer"
Jun 15 06:15:23.083018 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.082868    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0199cf99-cd82-48ca-924c-d6cb31ef6082" containerName="storage-initializer"
Jun 15 06:15:23.083018 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.082876    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0199cf99-cd82-48ca-924c-d6cb31ef6082" containerName="kube-rbac-proxy"
Jun 15 06:15:23.083018 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.082882    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0199cf99-cd82-48ca-924c-d6cb31ef6082" containerName="kube-rbac-proxy"
Jun 15 06:15:23.083018 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.082957    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0199cf99-cd82-48ca-924c-d6cb31ef6082" containerName="kserve-container"
Jun 15 06:15:23.083018 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.082964    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0199cf99-cd82-48ca-924c-d6cb31ef6082" containerName="kube-rbac-proxy"
Jun 15 06:15:23.087717 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.087697    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"
Jun 15 06:15:23.089808 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.089791    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Jun 15 06:15:23.089888 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.089797    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Jun 15 06:15:23.097841 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.097813    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"]
Jun 15 06:15:23.192882 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.192848    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzvs7\" (UniqueName: \"kubernetes.io/projected/047eb1ae-8c45-4e6e-b326-50c7a43550ed-kube-api-access-lzvs7\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m\" (UID: \"047eb1ae-8c45-4e6e-b326-50c7a43550ed\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"
Jun 15 06:15:23.192882 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.192886    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/047eb1ae-8c45-4e6e-b326-50c7a43550ed-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m\" (UID: \"047eb1ae-8c45-4e6e-b326-50c7a43550ed\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"
Jun 15 06:15:23.193078 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.192946    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/047eb1ae-8c45-4e6e-b326-50c7a43550ed-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m\" (UID: \"047eb1ae-8c45-4e6e-b326-50c7a43550ed\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"
Jun 15 06:15:23.193078 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.192986    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/047eb1ae-8c45-4e6e-b326-50c7a43550ed-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m\" (UID: \"047eb1ae-8c45-4e6e-b326-50c7a43550ed\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"
Jun 15 06:15:23.193078 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.193022    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/047eb1ae-8c45-4e6e-b326-50c7a43550ed-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m\" (UID: \"047eb1ae-8c45-4e6e-b326-50c7a43550ed\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"
Jun 15 06:15:23.222029 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.221999    2572 generic.go:356] "Generic (PLEG): container finished" podID="9f123126-bb80-4b22-bc54-abd0cfad587f" containerID="ea2f20d0109d2dc4795dec884bbb7aa257424fa09eeac0c1f82b73a8908fd853" exitCode=2
Jun 15 06:15:23.222187 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.222053    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h" event={"ID":"9f123126-bb80-4b22-bc54-abd0cfad587f","Type":"ContainerDied","Data":"ea2f20d0109d2dc4795dec884bbb7aa257424fa09eeac0c1f82b73a8908fd853"}
Jun 15 06:15:23.294498 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.294403    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzvs7\" (UniqueName: \"kubernetes.io/projected/047eb1ae-8c45-4e6e-b326-50c7a43550ed-kube-api-access-lzvs7\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m\" (UID: \"047eb1ae-8c45-4e6e-b326-50c7a43550ed\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"
Jun 15 06:15:23.294498 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.294450    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/047eb1ae-8c45-4e6e-b326-50c7a43550ed-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m\" (UID: \"047eb1ae-8c45-4e6e-b326-50c7a43550ed\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"
Jun 15 06:15:23.294726 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.294518    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/047eb1ae-8c45-4e6e-b326-50c7a43550ed-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m\" (UID: \"047eb1ae-8c45-4e6e-b326-50c7a43550ed\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"
Jun 15 06:15:23.294726 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.294565    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/047eb1ae-8c45-4e6e-b326-50c7a43550ed-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m\" (UID: \"047eb1ae-8c45-4e6e-b326-50c7a43550ed\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"
Jun 15 06:15:23.294726 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.294619    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/047eb1ae-8c45-4e6e-b326-50c7a43550ed-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m\" (UID: \"047eb1ae-8c45-4e6e-b326-50c7a43550ed\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"
Jun 15 06:15:23.294877 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:15:23.294765    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-serving-cert: secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Jun 15 06:15:23.294877 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:15:23.294837    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/047eb1ae-8c45-4e6e-b326-50c7a43550ed-proxy-tls podName:047eb1ae-8c45-4e6e-b326-50c7a43550ed nodeName:}" failed. No retries permitted until 2026-06-15 06:15:23.79481447 +0000 UTC m=+1638.942994468 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/047eb1ae-8c45-4e6e-b326-50c7a43550ed-proxy-tls") pod "isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m" (UID: "047eb1ae-8c45-4e6e-b326-50c7a43550ed") : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Jun 15 06:15:23.294998 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.294922    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/047eb1ae-8c45-4e6e-b326-50c7a43550ed-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m\" (UID: \"047eb1ae-8c45-4e6e-b326-50c7a43550ed\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"
Jun 15 06:15:23.295281 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.295262    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/047eb1ae-8c45-4e6e-b326-50c7a43550ed-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m\" (UID: \"047eb1ae-8c45-4e6e-b326-50c7a43550ed\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"
Jun 15 06:15:23.295386 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.295265    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/047eb1ae-8c45-4e6e-b326-50c7a43550ed-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m\" (UID: \"047eb1ae-8c45-4e6e-b326-50c7a43550ed\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"
Jun 15 06:15:23.303034 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.303010    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzvs7\" (UniqueName: \"kubernetes.io/projected/047eb1ae-8c45-4e6e-b326-50c7a43550ed-kube-api-access-lzvs7\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m\" (UID: \"047eb1ae-8c45-4e6e-b326-50c7a43550ed\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"
Jun 15 06:15:23.799478 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.799443    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/047eb1ae-8c45-4e6e-b326-50c7a43550ed-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m\" (UID: \"047eb1ae-8c45-4e6e-b326-50c7a43550ed\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"
Jun 15 06:15:23.801867 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.801835    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/047eb1ae-8c45-4e6e-b326-50c7a43550ed-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m\" (UID: \"047eb1ae-8c45-4e6e-b326-50c7a43550ed\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"
Jun 15 06:15:23.999087 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:23.999048    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"
Jun 15 06:15:24.131097 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:24.131072    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"]
Jun 15 06:15:24.133097 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:15:24.133070    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047eb1ae_8c45_4e6e_b326_50c7a43550ed.slice/crio-5805bda5bccac4aae63db2522bc6c483bbe3418e4d3365cc0c30b6a65bfb75bd WatchSource:0}: Error finding container 5805bda5bccac4aae63db2522bc6c483bbe3418e4d3365cc0c30b6a65bfb75bd: Status 404 returned error can't find the container with id 5805bda5bccac4aae63db2522bc6c483bbe3418e4d3365cc0c30b6a65bfb75bd
Jun 15 06:15:24.228718 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:24.228684    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m" event={"ID":"047eb1ae-8c45-4e6e-b326-50c7a43550ed","Type":"ContainerStarted","Data":"861e5b76f23e6726221e855a0b836146b320f98e8ffb2a95fc765d5e1195e360"}
Jun 15 06:15:24.228879 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:24.228759    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m" event={"ID":"047eb1ae-8c45-4e6e-b326-50c7a43550ed","Type":"ContainerStarted","Data":"5805bda5bccac4aae63db2522bc6c483bbe3418e4d3365cc0c30b6a65bfb75bd"}
Jun 15 06:15:24.970263 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:24.970219    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h" podUID="9f123126-bb80-4b22-bc54-abd0cfad587f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Jun 15 06:15:24.974497 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:24.974462    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h" podUID="9f123126-bb80-4b22-bc54-abd0cfad587f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jun 15 06:15:25.738685 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:25.738658    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h"
Jun 15 06:15:25.815311 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:25.815231    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f123126-bb80-4b22-bc54-abd0cfad587f-kserve-provision-location\") pod \"9f123126-bb80-4b22-bc54-abd0cfad587f\" (UID: \"9f123126-bb80-4b22-bc54-abd0cfad587f\") "
Jun 15 06:15:25.815311 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:25.815290    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f123126-bb80-4b22-bc54-abd0cfad587f-proxy-tls\") pod \"9f123126-bb80-4b22-bc54-abd0cfad587f\" (UID: \"9f123126-bb80-4b22-bc54-abd0cfad587f\") "
Jun 15 06:15:25.815311 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:25.815311    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f123126-bb80-4b22-bc54-abd0cfad587f-kube-rbac-proxy-sar-config\") pod \"9f123126-bb80-4b22-bc54-abd0cfad587f\" (UID: \"9f123126-bb80-4b22-bc54-abd0cfad587f\") "
Jun 15 06:15:25.815605 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:25.815351    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2bn2\" (UniqueName: \"kubernetes.io/projected/9f123126-bb80-4b22-bc54-abd0cfad587f-kube-api-access-x2bn2\") pod \"9f123126-bb80-4b22-bc54-abd0cfad587f\" (UID: \"9f123126-bb80-4b22-bc54-abd0cfad587f\") "
Jun 15 06:15:25.815605 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:25.815419    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f123126-bb80-4b22-bc54-abd0cfad587f-cabundle-cert\") pod \"9f123126-bb80-4b22-bc54-abd0cfad587f\" (UID: \"9f123126-bb80-4b22-bc54-abd0cfad587f\") "
Jun 15 06:15:25.816183 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:25.816127    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f123126-bb80-4b22-bc54-abd0cfad587f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9f123126-bb80-4b22-bc54-abd0cfad587f" (UID: "9f123126-bb80-4b22-bc54-abd0cfad587f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:15:25.822171 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:25.816484    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9f123126-bb80-4b22-bc54-abd0cfad587f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:15:25.822171 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:25.816562    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f123126-bb80-4b22-bc54-abd0cfad587f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9f123126-bb80-4b22-bc54-abd0cfad587f" (UID: "9f123126-bb80-4b22-bc54-abd0cfad587f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:15:25.823753 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:25.823722    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f123126-bb80-4b22-bc54-abd0cfad587f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9f123126-bb80-4b22-bc54-abd0cfad587f" (UID: "9f123126-bb80-4b22-bc54-abd0cfad587f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:15:25.823897 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:25.823879    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f123126-bb80-4b22-bc54-abd0cfad587f-kube-api-access-x2bn2" (OuterVolumeSpecName: "kube-api-access-x2bn2") pod "9f123126-bb80-4b22-bc54-abd0cfad587f" (UID: "9f123126-bb80-4b22-bc54-abd0cfad587f"). InnerVolumeSpecName "kube-api-access-x2bn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:15:25.828580 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:25.828526    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f123126-bb80-4b22-bc54-abd0cfad587f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9f123126-bb80-4b22-bc54-abd0cfad587f" (UID: "9f123126-bb80-4b22-bc54-abd0cfad587f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:15:25.917539 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:25.917506    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9f123126-bb80-4b22-bc54-abd0cfad587f-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:15:25.917539 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:25.917537    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9f123126-bb80-4b22-bc54-abd0cfad587f-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:15:25.917736 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:25.917550    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2bn2\" (UniqueName: \"kubernetes.io/projected/9f123126-bb80-4b22-bc54-abd0cfad587f-kube-api-access-x2bn2\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:15:25.917736 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:25.917560    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9f123126-bb80-4b22-bc54-abd0cfad587f-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:15:26.239866 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:26.239833    2572 generic.go:356] "Generic (PLEG): container finished" podID="9f123126-bb80-4b22-bc54-abd0cfad587f" containerID="f29f9c10bb09242d3f29e84b6e1227496cbeea4dec2638816536d14aec1c00a1" exitCode=0
Jun 15 06:15:26.240040 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:26.239889    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h" event={"ID":"9f123126-bb80-4b22-bc54-abd0cfad587f","Type":"ContainerDied","Data":"f29f9c10bb09242d3f29e84b6e1227496cbeea4dec2638816536d14aec1c00a1"}
Jun 15 06:15:26.240040 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:26.239902    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h"
Jun 15 06:15:26.240040 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:26.239964    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h" event={"ID":"9f123126-bb80-4b22-bc54-abd0cfad587f","Type":"ContainerDied","Data":"2dd3e10df63afcd0e70b36a334e7bce973c0c9eddcb1a12e0165931a07a882cc"}
Jun 15 06:15:26.240040 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:26.239990    2572 scope.go:117] "RemoveContainer" containerID="ea2f20d0109d2dc4795dec884bbb7aa257424fa09eeac0c1f82b73a8908fd853"
Jun 15 06:15:26.249105 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:26.249085    2572 scope.go:117] "RemoveContainer" containerID="f29f9c10bb09242d3f29e84b6e1227496cbeea4dec2638816536d14aec1c00a1"
Jun 15 06:15:26.257181 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:26.257165    2572 scope.go:117] "RemoveContainer" containerID="b7d9283a3669e7c26bbb39b9334fee31c8544622fe2a5c06792d062bcd651700"
Jun 15 06:15:26.264714 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:26.264694    2572 scope.go:117] "RemoveContainer" containerID="ea2f20d0109d2dc4795dec884bbb7aa257424fa09eeac0c1f82b73a8908fd853"
Jun 15 06:15:26.264978 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:15:26.264960    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea2f20d0109d2dc4795dec884bbb7aa257424fa09eeac0c1f82b73a8908fd853\": container with ID starting with ea2f20d0109d2dc4795dec884bbb7aa257424fa09eeac0c1f82b73a8908fd853 not found: ID does not exist" containerID="ea2f20d0109d2dc4795dec884bbb7aa257424fa09eeac0c1f82b73a8908fd853"
Jun 15 06:15:26.265044 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:26.264994    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea2f20d0109d2dc4795dec884bbb7aa257424fa09eeac0c1f82b73a8908fd853"} err="failed to get container status \"ea2f20d0109d2dc4795dec884bbb7aa257424fa09eeac0c1f82b73a8908fd853\": rpc error: code = NotFound desc = could not find container \"ea2f20d0109d2dc4795dec884bbb7aa257424fa09eeac0c1f82b73a8908fd853\": container with ID starting with ea2f20d0109d2dc4795dec884bbb7aa257424fa09eeac0c1f82b73a8908fd853 not found: ID does not exist"
Jun 15 06:15:26.265044 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:26.265011    2572 scope.go:117] "RemoveContainer" containerID="f29f9c10bb09242d3f29e84b6e1227496cbeea4dec2638816536d14aec1c00a1"
Jun 15 06:15:26.265190 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:26.265171    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h"]
Jun 15 06:15:26.265238 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:15:26.265226    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f29f9c10bb09242d3f29e84b6e1227496cbeea4dec2638816536d14aec1c00a1\": container with ID starting with f29f9c10bb09242d3f29e84b6e1227496cbeea4dec2638816536d14aec1c00a1 not found: ID does not exist" containerID="f29f9c10bb09242d3f29e84b6e1227496cbeea4dec2638816536d14aec1c00a1"
Jun 15 06:15:26.265276 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:26.265247    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f29f9c10bb09242d3f29e84b6e1227496cbeea4dec2638816536d14aec1c00a1"} err="failed to get container status \"f29f9c10bb09242d3f29e84b6e1227496cbeea4dec2638816536d14aec1c00a1\": rpc error: code = NotFound desc = could not find container \"f29f9c10bb09242d3f29e84b6e1227496cbeea4dec2638816536d14aec1c00a1\": container with ID starting with f29f9c10bb09242d3f29e84b6e1227496cbeea4dec2638816536d14aec1c00a1 not found: ID does not exist"
Jun 15 06:15:26.265276 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:26.265262    2572 scope.go:117] "RemoveContainer" containerID="b7d9283a3669e7c26bbb39b9334fee31c8544622fe2a5c06792d062bcd651700"
Jun 15 06:15:26.265486 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:15:26.265469    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7d9283a3669e7c26bbb39b9334fee31c8544622fe2a5c06792d062bcd651700\": container with ID starting with b7d9283a3669e7c26bbb39b9334fee31c8544622fe2a5c06792d062bcd651700 not found: ID does not exist" containerID="b7d9283a3669e7c26bbb39b9334fee31c8544622fe2a5c06792d062bcd651700"
Jun 15 06:15:26.265547 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:26.265489    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7d9283a3669e7c26bbb39b9334fee31c8544622fe2a5c06792d062bcd651700"} err="failed to get container status \"b7d9283a3669e7c26bbb39b9334fee31c8544622fe2a5c06792d062bcd651700\": rpc error: code = NotFound desc = could not find container \"b7d9283a3669e7c26bbb39b9334fee31c8544622fe2a5c06792d062bcd651700\": container with ID starting with b7d9283a3669e7c26bbb39b9334fee31c8544622fe2a5c06792d062bcd651700 not found: ID does not exist"
Jun 15 06:15:26.269485 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:26.269465    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-jrz5h"]
Jun 15 06:15:27.323432 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:27.323403    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f123126-bb80-4b22-bc54-abd0cfad587f" path="/var/lib/kubelet/pods/9f123126-bb80-4b22-bc54-abd0cfad587f/volumes"
Jun 15 06:15:29.256661 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:29.256628    2572 generic.go:356] "Generic (PLEG): container finished" podID="047eb1ae-8c45-4e6e-b326-50c7a43550ed" containerID="861e5b76f23e6726221e855a0b836146b320f98e8ffb2a95fc765d5e1195e360" exitCode=0
Jun 15 06:15:29.257146 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:29.256706    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m" event={"ID":"047eb1ae-8c45-4e6e-b326-50c7a43550ed","Type":"ContainerDied","Data":"861e5b76f23e6726221e855a0b836146b320f98e8ffb2a95fc765d5e1195e360"}
Jun 15 06:15:30.263739 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:30.263698    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m" event={"ID":"047eb1ae-8c45-4e6e-b326-50c7a43550ed","Type":"ContainerStarted","Data":"cbeb334c787cdb576f22e5228fdd47c8408e3d2e9f268d7940b203f9c7c9c660"}
Jun 15 06:15:30.264109 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:30.263759    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m" event={"ID":"047eb1ae-8c45-4e6e-b326-50c7a43550ed","Type":"ContainerStarted","Data":"87a85abac08f8444e20bfae10670c0564476a6754e3e42a67bd004b20df0718e"}
Jun 15 06:15:30.264109 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:30.264029    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"
Jun 15 06:15:30.264206 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:30.264167    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"
Jun 15 06:15:30.265489 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:30.265462    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m" podUID="047eb1ae-8c45-4e6e-b326-50c7a43550ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jun 15 06:15:30.295039 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:30.294994    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m" podStartSLOduration=2.823917338 podStartE2EDuration="7.294979239s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.471061901s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:15:30.294786233 +0000 UTC m=+1645.442966242" watchObservedRunningTime="2026-06-15 06:15:30.294979239 +0000 UTC m=+1645.443159235"
Jun 15 06:15:31.268788 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:31.268747    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m" podUID="047eb1ae-8c45-4e6e-b326-50c7a43550ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jun 15 06:15:36.274434 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:36.274406    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"
Jun 15 06:15:36.274910 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:36.274883    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m" podUID="047eb1ae-8c45-4e6e-b326-50c7a43550ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jun 15 06:15:46.275332 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:46.275289    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m" podUID="047eb1ae-8c45-4e6e-b326-50c7a43550ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jun 15 06:15:56.275282 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:15:56.275243    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m" podUID="047eb1ae-8c45-4e6e-b326-50c7a43550ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jun 15 06:16:06.275087 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:06.275051    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m" podUID="047eb1ae-8c45-4e6e-b326-50c7a43550ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jun 15 06:16:16.275561 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:16.275493    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"
Jun 15 06:16:24.475019 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.474987    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"]
Jun 15 06:16:24.475543 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.475220    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m" podUID="047eb1ae-8c45-4e6e-b326-50c7a43550ed" containerName="kserve-container" containerID="cri-o://87a85abac08f8444e20bfae10670c0564476a6754e3e42a67bd004b20df0718e" gracePeriod=30
Jun 15 06:16:24.475543 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.475263    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m" podUID="047eb1ae-8c45-4e6e-b326-50c7a43550ed" containerName="kube-rbac-proxy" containerID="cri-o://cbeb334c787cdb576f22e5228fdd47c8408e3d2e9f268d7940b203f9c7c9c660" gracePeriod=30
Jun 15 06:16:24.556213 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.556177    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"]
Jun 15 06:16:24.556574 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.556560    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f123126-bb80-4b22-bc54-abd0cfad587f" containerName="kube-rbac-proxy"
Jun 15 06:16:24.556574 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.556575    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f123126-bb80-4b22-bc54-abd0cfad587f" containerName="kube-rbac-proxy"
Jun 15 06:16:24.556712 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.556587    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f123126-bb80-4b22-bc54-abd0cfad587f" containerName="kserve-container"
Jun 15 06:16:24.556712 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.556592    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f123126-bb80-4b22-bc54-abd0cfad587f" containerName="kserve-container"
Jun 15 06:16:24.556712 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.556607    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f123126-bb80-4b22-bc54-abd0cfad587f" containerName="storage-initializer"
Jun 15 06:16:24.556712 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.556613    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f123126-bb80-4b22-bc54-abd0cfad587f" containerName="storage-initializer"
Jun 15 06:16:24.556712 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.556670    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f123126-bb80-4b22-bc54-abd0cfad587f" containerName="kube-rbac-proxy"
Jun 15 06:16:24.556712 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.556678    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f123126-bb80-4b22-bc54-abd0cfad587f" containerName="kserve-container"
Jun 15 06:16:24.561462 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.561443    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"
Jun 15 06:16:24.563416 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.563392    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Jun 15 06:16:24.563537 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.563413    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Jun 15 06:16:24.569733 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.569711    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"]
Jun 15 06:16:24.638598 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.638563    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-4wb7h\" (UID: \"7d0e2456-d3c7-4776-a6b8-644cff2e21e1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"
Jun 15 06:16:24.638765 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.638610    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-4wb7h\" (UID: \"7d0e2456-d3c7-4776-a6b8-644cff2e21e1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"
Jun 15 06:16:24.638765 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.638671    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-4wb7h\" (UID: \"7d0e2456-d3c7-4776-a6b8-644cff2e21e1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"
Jun 15 06:16:24.638765 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.638711    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pcsn\" (UniqueName: \"kubernetes.io/projected/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-kube-api-access-2pcsn\") pod \"isvc-pmml-predictor-8674795f59-4wb7h\" (UID: \"7d0e2456-d3c7-4776-a6b8-644cff2e21e1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"
Jun 15 06:16:24.638913 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.638767    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-4wb7h\" (UID: \"7d0e2456-d3c7-4776-a6b8-644cff2e21e1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"
Jun 15 06:16:24.740349 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.740240    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-4wb7h\" (UID: \"7d0e2456-d3c7-4776-a6b8-644cff2e21e1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"
Jun 15 06:16:24.740349 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.740313    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-4wb7h\" (UID: \"7d0e2456-d3c7-4776-a6b8-644cff2e21e1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"
Jun 15 06:16:24.740660 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.740384    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pcsn\" (UniqueName: \"kubernetes.io/projected/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-kube-api-access-2pcsn\") pod \"isvc-pmml-predictor-8674795f59-4wb7h\" (UID: \"7d0e2456-d3c7-4776-a6b8-644cff2e21e1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"
Jun 15 06:16:24.740660 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.740457    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-4wb7h\" (UID: \"7d0e2456-d3c7-4776-a6b8-644cff2e21e1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"
Jun 15 06:16:24.740660 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.740534    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-4wb7h\" (UID: \"7d0e2456-d3c7-4776-a6b8-644cff2e21e1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"
Jun 15 06:16:24.740660 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:16:24.740642    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-predictor-serving-cert: secret "isvc-pmml-predictor-serving-cert" not found
Jun 15 06:16:24.740855 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:16:24.740726    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-proxy-tls podName:7d0e2456-d3c7-4776-a6b8-644cff2e21e1 nodeName:}" failed. No retries permitted until 2026-06-15 06:16:25.240703324 +0000 UTC m=+1700.388883325 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-proxy-tls") pod "isvc-pmml-predictor-8674795f59-4wb7h" (UID: "7d0e2456-d3c7-4776-a6b8-644cff2e21e1") : secret "isvc-pmml-predictor-serving-cert" not found
Jun 15 06:16:24.740930 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.740910    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-4wb7h\" (UID: \"7d0e2456-d3c7-4776-a6b8-644cff2e21e1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"
Jun 15 06:16:24.741113 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.741092    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-4wb7h\" (UID: \"7d0e2456-d3c7-4776-a6b8-644cff2e21e1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"
Jun 15 06:16:24.741154 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.741092    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-4wb7h\" (UID: \"7d0e2456-d3c7-4776-a6b8-644cff2e21e1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"
Jun 15 06:16:24.751633 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:24.751608    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pcsn\" (UniqueName: \"kubernetes.io/projected/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-kube-api-access-2pcsn\") pod \"isvc-pmml-predictor-8674795f59-4wb7h\" (UID: \"7d0e2456-d3c7-4776-a6b8-644cff2e21e1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"
Jun 15 06:16:25.245474 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:25.245442    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-4wb7h\" (UID: \"7d0e2456-d3c7-4776-a6b8-644cff2e21e1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"
Jun 15 06:16:25.248010 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:25.247978    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-4wb7h\" (UID: \"7d0e2456-d3c7-4776-a6b8-644cff2e21e1\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"
Jun 15 06:16:25.472923 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:25.472884    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"
Jun 15 06:16:25.518378 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:25.518278    2572 generic.go:356] "Generic (PLEG): container finished" podID="047eb1ae-8c45-4e6e-b326-50c7a43550ed" containerID="cbeb334c787cdb576f22e5228fdd47c8408e3d2e9f268d7940b203f9c7c9c660" exitCode=2
Jun 15 06:16:25.518378 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:25.518350    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m" event={"ID":"047eb1ae-8c45-4e6e-b326-50c7a43550ed","Type":"ContainerDied","Data":"cbeb334c787cdb576f22e5228fdd47c8408e3d2e9f268d7940b203f9c7c9c660"}
Jun 15 06:16:25.608740 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:25.608719    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"]
Jun 15 06:16:25.610803 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:16:25.610779    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d0e2456_d3c7_4776_a6b8_644cff2e21e1.slice/crio-263b20048985d22b1e81f0a586d0bec88821a076b0b55ee6a5fcbee58ad64ca6 WatchSource:0}: Error finding container 263b20048985d22b1e81f0a586d0bec88821a076b0b55ee6a5fcbee58ad64ca6: Status 404 returned error can't find the container with id 263b20048985d22b1e81f0a586d0bec88821a076b0b55ee6a5fcbee58ad64ca6
Jun 15 06:16:26.269468 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:26.269425    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m" podUID="047eb1ae-8c45-4e6e-b326-50c7a43550ed" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Jun 15 06:16:26.275178 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:26.275152    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m" podUID="047eb1ae-8c45-4e6e-b326-50c7a43550ed" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jun 15 06:16:26.524083 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:26.523992    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h" event={"ID":"7d0e2456-d3c7-4776-a6b8-644cff2e21e1","Type":"ContainerStarted","Data":"25d4abac1b52e9b7404b595ffc8dae8e6154be64b64de82613e44e004dd23b5d"}
Jun 15 06:16:26.524083 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:26.524051    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h" event={"ID":"7d0e2456-d3c7-4776-a6b8-644cff2e21e1","Type":"ContainerStarted","Data":"263b20048985d22b1e81f0a586d0bec88821a076b0b55ee6a5fcbee58ad64ca6"}
Jun 15 06:16:27.217032 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.217005    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"
Jun 15 06:16:27.367702 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.367612    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/047eb1ae-8c45-4e6e-b326-50c7a43550ed-kube-rbac-proxy-sar-config\") pod \"047eb1ae-8c45-4e6e-b326-50c7a43550ed\" (UID: \"047eb1ae-8c45-4e6e-b326-50c7a43550ed\") "
Jun 15 06:16:27.367880 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.367719    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/047eb1ae-8c45-4e6e-b326-50c7a43550ed-cabundle-cert\") pod \"047eb1ae-8c45-4e6e-b326-50c7a43550ed\" (UID: \"047eb1ae-8c45-4e6e-b326-50c7a43550ed\") "
Jun 15 06:16:27.367880 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.367748    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/047eb1ae-8c45-4e6e-b326-50c7a43550ed-kserve-provision-location\") pod \"047eb1ae-8c45-4e6e-b326-50c7a43550ed\" (UID: \"047eb1ae-8c45-4e6e-b326-50c7a43550ed\") "
Jun 15 06:16:27.367880 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.367773    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzvs7\" (UniqueName: \"kubernetes.io/projected/047eb1ae-8c45-4e6e-b326-50c7a43550ed-kube-api-access-lzvs7\") pod \"047eb1ae-8c45-4e6e-b326-50c7a43550ed\" (UID: \"047eb1ae-8c45-4e6e-b326-50c7a43550ed\") "
Jun 15 06:16:27.367880 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.367831    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/047eb1ae-8c45-4e6e-b326-50c7a43550ed-proxy-tls\") pod \"047eb1ae-8c45-4e6e-b326-50c7a43550ed\" (UID: \"047eb1ae-8c45-4e6e-b326-50c7a43550ed\") "
Jun 15 06:16:27.368099 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.368061    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/047eb1ae-8c45-4e6e-b326-50c7a43550ed-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "047eb1ae-8c45-4e6e-b326-50c7a43550ed" (UID: "047eb1ae-8c45-4e6e-b326-50c7a43550ed"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:16:27.368099 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.368075    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/047eb1ae-8c45-4e6e-b326-50c7a43550ed-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "047eb1ae-8c45-4e6e-b326-50c7a43550ed" (UID: "047eb1ae-8c45-4e6e-b326-50c7a43550ed"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:16:27.369984 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.369945    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/047eb1ae-8c45-4e6e-b326-50c7a43550ed-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "047eb1ae-8c45-4e6e-b326-50c7a43550ed" (UID: "047eb1ae-8c45-4e6e-b326-50c7a43550ed"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:16:27.370099 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.370023    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/047eb1ae-8c45-4e6e-b326-50c7a43550ed-kube-api-access-lzvs7" (OuterVolumeSpecName: "kube-api-access-lzvs7") pod "047eb1ae-8c45-4e6e-b326-50c7a43550ed" (UID: "047eb1ae-8c45-4e6e-b326-50c7a43550ed"). InnerVolumeSpecName "kube-api-access-lzvs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:16:27.377600 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.377573    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/047eb1ae-8c45-4e6e-b326-50c7a43550ed-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "047eb1ae-8c45-4e6e-b326-50c7a43550ed" (UID: "047eb1ae-8c45-4e6e-b326-50c7a43550ed"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:16:27.468619 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.468574    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/047eb1ae-8c45-4e6e-b326-50c7a43550ed-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:16:27.468619 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.468615    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/047eb1ae-8c45-4e6e-b326-50c7a43550ed-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:16:27.468619 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.468626    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzvs7\" (UniqueName: \"kubernetes.io/projected/047eb1ae-8c45-4e6e-b326-50c7a43550ed-kube-api-access-lzvs7\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:16:27.468619 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.468635    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/047eb1ae-8c45-4e6e-b326-50c7a43550ed-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:16:27.468915 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.468645    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/047eb1ae-8c45-4e6e-b326-50c7a43550ed-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:16:27.530528 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.530492    2572 generic.go:356] "Generic (PLEG): container finished" podID="047eb1ae-8c45-4e6e-b326-50c7a43550ed" containerID="87a85abac08f8444e20bfae10670c0564476a6754e3e42a67bd004b20df0718e" exitCode=0
Jun 15 06:16:27.530933 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.530548    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"
Jun 15 06:16:27.530933 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.530581    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m" event={"ID":"047eb1ae-8c45-4e6e-b326-50c7a43550ed","Type":"ContainerDied","Data":"87a85abac08f8444e20bfae10670c0564476a6754e3e42a67bd004b20df0718e"}
Jun 15 06:16:27.530933 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.530638    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m" event={"ID":"047eb1ae-8c45-4e6e-b326-50c7a43550ed","Type":"ContainerDied","Data":"5805bda5bccac4aae63db2522bc6c483bbe3418e4d3365cc0c30b6a65bfb75bd"}
Jun 15 06:16:27.530933 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.530663    2572 scope.go:117] "RemoveContainer" containerID="cbeb334c787cdb576f22e5228fdd47c8408e3d2e9f268d7940b203f9c7c9c660"
Jun 15 06:16:27.542269 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.542251    2572 scope.go:117] "RemoveContainer" containerID="87a85abac08f8444e20bfae10670c0564476a6754e3e42a67bd004b20df0718e"
Jun 15 06:16:27.550089 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.550070    2572 scope.go:117] "RemoveContainer" containerID="861e5b76f23e6726221e855a0b836146b320f98e8ffb2a95fc765d5e1195e360"
Jun 15 06:16:27.556454 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.556433    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"]
Jun 15 06:16:27.557772 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.557760    2572 scope.go:117] "RemoveContainer" containerID="cbeb334c787cdb576f22e5228fdd47c8408e3d2e9f268d7940b203f9c7c9c660"
Jun 15 06:16:27.558014 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:16:27.557991    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbeb334c787cdb576f22e5228fdd47c8408e3d2e9f268d7940b203f9c7c9c660\": container with ID starting with cbeb334c787cdb576f22e5228fdd47c8408e3d2e9f268d7940b203f9c7c9c660 not found: ID does not exist" containerID="cbeb334c787cdb576f22e5228fdd47c8408e3d2e9f268d7940b203f9c7c9c660"
Jun 15 06:16:27.558113 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.558021    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbeb334c787cdb576f22e5228fdd47c8408e3d2e9f268d7940b203f9c7c9c660"} err="failed to get container status \"cbeb334c787cdb576f22e5228fdd47c8408e3d2e9f268d7940b203f9c7c9c660\": rpc error: code = NotFound desc = could not find container \"cbeb334c787cdb576f22e5228fdd47c8408e3d2e9f268d7940b203f9c7c9c660\": container with ID starting with cbeb334c787cdb576f22e5228fdd47c8408e3d2e9f268d7940b203f9c7c9c660 not found: ID does not exist"
Jun 15 06:16:27.558113 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.558048    2572 scope.go:117] "RemoveContainer" containerID="87a85abac08f8444e20bfae10670c0564476a6754e3e42a67bd004b20df0718e"
Jun 15 06:16:27.558303 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:16:27.558285    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87a85abac08f8444e20bfae10670c0564476a6754e3e42a67bd004b20df0718e\": container with ID starting with 87a85abac08f8444e20bfae10670c0564476a6754e3e42a67bd004b20df0718e not found: ID does not exist" containerID="87a85abac08f8444e20bfae10670c0564476a6754e3e42a67bd004b20df0718e"
Jun 15 06:16:27.558345 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.558310    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87a85abac08f8444e20bfae10670c0564476a6754e3e42a67bd004b20df0718e"} err="failed to get container status \"87a85abac08f8444e20bfae10670c0564476a6754e3e42a67bd004b20df0718e\": rpc error: code = NotFound desc = could not find container \"87a85abac08f8444e20bfae10670c0564476a6754e3e42a67bd004b20df0718e\": container with ID starting with 87a85abac08f8444e20bfae10670c0564476a6754e3e42a67bd004b20df0718e not found: ID does not exist"
Jun 15 06:16:27.558345 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.558325    2572 scope.go:117] "RemoveContainer" containerID="861e5b76f23e6726221e855a0b836146b320f98e8ffb2a95fc765d5e1195e360"
Jun 15 06:16:27.558611 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:16:27.558595    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"861e5b76f23e6726221e855a0b836146b320f98e8ffb2a95fc765d5e1195e360\": container with ID starting with 861e5b76f23e6726221e855a0b836146b320f98e8ffb2a95fc765d5e1195e360 not found: ID does not exist" containerID="861e5b76f23e6726221e855a0b836146b320f98e8ffb2a95fc765d5e1195e360"
Jun 15 06:16:27.558666 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.558616    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"861e5b76f23e6726221e855a0b836146b320f98e8ffb2a95fc765d5e1195e360"} err="failed to get container status \"861e5b76f23e6726221e855a0b836146b320f98e8ffb2a95fc765d5e1195e360\": rpc error: code = NotFound desc = could not find container \"861e5b76f23e6726221e855a0b836146b320f98e8ffb2a95fc765d5e1195e360\": container with ID starting with 861e5b76f23e6726221e855a0b836146b320f98e8ffb2a95fc765d5e1195e360 not found: ID does not exist"
Jun 15 06:16:27.563422 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:27.563403    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-fwm5m"]
Jun 15 06:16:29.323981 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:29.323944    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="047eb1ae-8c45-4e6e-b326-50c7a43550ed" path="/var/lib/kubelet/pods/047eb1ae-8c45-4e6e-b326-50c7a43550ed/volumes"
Jun 15 06:16:29.542990 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:29.542957    2572 generic.go:356] "Generic (PLEG): container finished" podID="7d0e2456-d3c7-4776-a6b8-644cff2e21e1" containerID="25d4abac1b52e9b7404b595ffc8dae8e6154be64b64de82613e44e004dd23b5d" exitCode=0
Jun 15 06:16:29.543167 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:29.543033    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h" event={"ID":"7d0e2456-d3c7-4776-a6b8-644cff2e21e1","Type":"ContainerDied","Data":"25d4abac1b52e9b7404b595ffc8dae8e6154be64b64de82613e44e004dd23b5d"}
Jun 15 06:16:37.593763 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:37.593722    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h" event={"ID":"7d0e2456-d3c7-4776-a6b8-644cff2e21e1","Type":"ContainerStarted","Data":"830e9a1660128af09ae3420736b23d00d34863658154503e1cfb2c42e22a4451"}
Jun 15 06:16:37.594202 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:37.593796    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h" event={"ID":"7d0e2456-d3c7-4776-a6b8-644cff2e21e1","Type":"ContainerStarted","Data":"e13176016adbe81f06bcbf08a3f0e7348b531ef120feb32249f0b27196bcf035"}
Jun 15 06:16:37.594202 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:37.594089    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"
Jun 15 06:16:37.594310 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:37.594243    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"
Jun 15 06:16:37.595299 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:37.595273    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h" podUID="7d0e2456-d3c7-4776-a6b8-644cff2e21e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jun 15 06:16:37.615409 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:37.615347    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h" podStartSLOduration=2.547001344 podStartE2EDuration="13.615334428s" totalImagesPullingTime="7.322620897s" totalInitContainerRuntime="3.745712187s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:16:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:16:37.6130194 +0000 UTC m=+1712.761199397" watchObservedRunningTime="2026-06-15 06:16:37.615334428 +0000 UTC m=+1712.763514423"
Jun 15 06:16:38.598810 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:38.598771    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h" podUID="7d0e2456-d3c7-4776-a6b8-644cff2e21e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jun 15 06:16:43.603758 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:43.603727    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"
Jun 15 06:16:43.604235 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:43.604210    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h" podUID="7d0e2456-d3c7-4776-a6b8-644cff2e21e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jun 15 06:16:53.604657 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:16:53.604615    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h" podUID="7d0e2456-d3c7-4776-a6b8-644cff2e21e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jun 15 06:17:03.604353 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:17:03.604308    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h" podUID="7d0e2456-d3c7-4776-a6b8-644cff2e21e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jun 15 06:17:13.604944 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:17:13.604903    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h" podUID="7d0e2456-d3c7-4776-a6b8-644cff2e21e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jun 15 06:17:23.605135 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:17:23.605093    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h" podUID="7d0e2456-d3c7-4776-a6b8-644cff2e21e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jun 15 06:17:33.604810 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:17:33.604768    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h" podUID="7d0e2456-d3c7-4776-a6b8-644cff2e21e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jun 15 06:17:43.605181 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:17:43.605139    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h" podUID="7d0e2456-d3c7-4776-a6b8-644cff2e21e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jun 15 06:17:45.321214 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:17:45.321133    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h" podUID="7d0e2456-d3c7-4776-a6b8-644cff2e21e1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jun 15 06:17:55.324001 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:17:55.323972    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"
Jun 15 06:18:05.421183 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:05.421151    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 06:18:05.429796 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:05.429775    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 06:18:05.800131 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:05.800085    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"]
Jun 15 06:18:05.800424 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:05.800388    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h" podUID="7d0e2456-d3c7-4776-a6b8-644cff2e21e1" containerName="kserve-container" containerID="cri-o://e13176016adbe81f06bcbf08a3f0e7348b531ef120feb32249f0b27196bcf035" gracePeriod=30
Jun 15 06:18:05.800572 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:05.800418    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h" podUID="7d0e2456-d3c7-4776-a6b8-644cff2e21e1" containerName="kube-rbac-proxy" containerID="cri-o://830e9a1660128af09ae3420736b23d00d34863658154503e1cfb2c42e22a4451" gracePeriod=30
Jun 15 06:18:05.935684 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:05.935655    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn"]
Jun 15 06:18:05.936040 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:05.936024    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="047eb1ae-8c45-4e6e-b326-50c7a43550ed" containerName="storage-initializer"
Jun 15 06:18:05.936040 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:05.936040    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="047eb1ae-8c45-4e6e-b326-50c7a43550ed" containerName="storage-initializer"
Jun 15 06:18:05.936171 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:05.936055    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="047eb1ae-8c45-4e6e-b326-50c7a43550ed" containerName="kserve-container"
Jun 15 06:18:05.936171 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:05.936061    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="047eb1ae-8c45-4e6e-b326-50c7a43550ed" containerName="kserve-container"
Jun 15 06:18:05.936171 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:05.936079    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="047eb1ae-8c45-4e6e-b326-50c7a43550ed" containerName="kube-rbac-proxy"
Jun 15 06:18:05.936171 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:05.936084    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="047eb1ae-8c45-4e6e-b326-50c7a43550ed" containerName="kube-rbac-proxy"
Jun 15 06:18:05.936171 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:05.936141    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="047eb1ae-8c45-4e6e-b326-50c7a43550ed" containerName="kserve-container"
Jun 15 06:18:05.936171 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:05.936157    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="047eb1ae-8c45-4e6e-b326-50c7a43550ed" containerName="kube-rbac-proxy"
Jun 15 06:18:05.939221 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:05.939205    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn"
Jun 15 06:18:05.941474 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:05.941446    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-predictor-serving-cert\""
Jun 15 06:18:05.941600 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:05.941484    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-runtime-kube-rbac-proxy-sar-config\""
Jun 15 06:18:05.949033 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:05.949008    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn"]
Jun 15 06:18:05.959476 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:05.959450    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-2xqrn\" (UID: \"b21d1f4a-8383-425f-a3c1-96c9e65dc9df\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn"
Jun 15 06:18:05.959591 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:05.959491    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-2xqrn\" (UID: \"b21d1f4a-8383-425f-a3c1-96c9e65dc9df\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn"
Jun 15 06:18:05.959591 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:05.959562    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-2xqrn\" (UID: \"b21d1f4a-8383-425f-a3c1-96c9e65dc9df\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn"
Jun 15 06:18:05.959685 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:05.959667    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxthn\" (UniqueName: \"kubernetes.io/projected/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-kube-api-access-wxthn\") pod \"isvc-pmml-runtime-predictor-885fc8777-2xqrn\" (UID: \"b21d1f4a-8383-425f-a3c1-96c9e65dc9df\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn"
Jun 15 06:18:05.959745 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:05.959722    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-2xqrn\" (UID: \"b21d1f4a-8383-425f-a3c1-96c9e65dc9df\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn"
Jun 15 06:18:06.013575 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:06.013547    2572 generic.go:356] "Generic (PLEG): container finished" podID="7d0e2456-d3c7-4776-a6b8-644cff2e21e1" containerID="830e9a1660128af09ae3420736b23d00d34863658154503e1cfb2c42e22a4451" exitCode=2
Jun 15 06:18:06.013746 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:06.013628    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h" event={"ID":"7d0e2456-d3c7-4776-a6b8-644cff2e21e1","Type":"ContainerDied","Data":"830e9a1660128af09ae3420736b23d00d34863658154503e1cfb2c42e22a4451"}
Jun 15 06:18:06.060225 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:06.060145    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-2xqrn\" (UID: \"b21d1f4a-8383-425f-a3c1-96c9e65dc9df\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn"
Jun 15 06:18:06.060225 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:06.060199    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-2xqrn\" (UID: \"b21d1f4a-8383-425f-a3c1-96c9e65dc9df\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn"
Jun 15 06:18:06.060225 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:06.060225    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-2xqrn\" (UID: \"b21d1f4a-8383-425f-a3c1-96c9e65dc9df\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn"
Jun 15 06:18:06.060494 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:06.060250    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-2xqrn\" (UID: \"b21d1f4a-8383-425f-a3c1-96c9e65dc9df\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn"
Jun 15 06:18:06.060494 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:06.060284    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxthn\" (UniqueName: \"kubernetes.io/projected/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-kube-api-access-wxthn\") pod \"isvc-pmml-runtime-predictor-885fc8777-2xqrn\" (UID: \"b21d1f4a-8383-425f-a3c1-96c9e65dc9df\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn"
Jun 15 06:18:06.060612 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:06.060529    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-kserve-provision-location\") pod \"isvc-pmml-runtime-predictor-885fc8777-2xqrn\" (UID: \"b21d1f4a-8383-425f-a3c1-96c9e65dc9df\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn"
Jun 15 06:18:06.061012 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:06.060990    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-cabundle-cert\") pod \"isvc-pmml-runtime-predictor-885fc8777-2xqrn\" (UID: \"b21d1f4a-8383-425f-a3c1-96c9e65dc9df\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn"
Jun 15 06:18:06.061081 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:06.061010    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-runtime-predictor-885fc8777-2xqrn\" (UID: \"b21d1f4a-8383-425f-a3c1-96c9e65dc9df\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn"
Jun 15 06:18:06.062801 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:06.062779    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-proxy-tls\") pod \"isvc-pmml-runtime-predictor-885fc8777-2xqrn\" (UID: \"b21d1f4a-8383-425f-a3c1-96c9e65dc9df\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn"
Jun 15 06:18:06.070929 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:06.070911    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxthn\" (UniqueName: \"kubernetes.io/projected/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-kube-api-access-wxthn\") pod \"isvc-pmml-runtime-predictor-885fc8777-2xqrn\" (UID: \"b21d1f4a-8383-425f-a3c1-96c9e65dc9df\") " pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn"
Jun 15 06:18:06.251409 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:06.251370    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn"
Jun 15 06:18:06.387219 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:06.387179    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn"]
Jun 15 06:18:06.391991 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:18:06.391960    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb21d1f4a_8383_425f_a3c1_96c9e65dc9df.slice/crio-4aba301478ec7ac193ead881a984cf0de85d1879744c0c7bb67ce044cf4d1855 WatchSource:0}: Error finding container 4aba301478ec7ac193ead881a984cf0de85d1879744c0c7bb67ce044cf4d1855: Status 404 returned error can't find the container with id 4aba301478ec7ac193ead881a984cf0de85d1879744c0c7bb67ce044cf4d1855
Jun 15 06:18:07.020940 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:07.020907    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn" event={"ID":"b21d1f4a-8383-425f-a3c1-96c9e65dc9df","Type":"ContainerStarted","Data":"9e25d8b9946ddcde8804e65781fba77f1e38b014735d16ac507b1d8c70753e2c"}
Jun 15 06:18:07.021340 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:07.020961    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn" event={"ID":"b21d1f4a-8383-425f-a3c1-96c9e65dc9df","Type":"ContainerStarted","Data":"4aba301478ec7ac193ead881a984cf0de85d1879744c0c7bb67ce044cf4d1855"}
Jun 15 06:18:08.599353 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:08.599307    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h" podUID="7d0e2456-d3c7-4776-a6b8-644cff2e21e1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.42:8643/healthz\": dial tcp 10.134.0.42:8643: connect: connection refused"
Jun 15 06:18:09.448878 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:09.448853    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"
Jun 15 06:18:09.490250 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:09.490186    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-cabundle-cert\") pod \"7d0e2456-d3c7-4776-a6b8-644cff2e21e1\" (UID: \"7d0e2456-d3c7-4776-a6b8-644cff2e21e1\") "
Jun 15 06:18:09.490250 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:09.490218    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-proxy-tls\") pod \"7d0e2456-d3c7-4776-a6b8-644cff2e21e1\" (UID: \"7d0e2456-d3c7-4776-a6b8-644cff2e21e1\") "
Jun 15 06:18:09.490435 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:09.490268    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-kube-rbac-proxy-sar-config\") pod \"7d0e2456-d3c7-4776-a6b8-644cff2e21e1\" (UID: \"7d0e2456-d3c7-4776-a6b8-644cff2e21e1\") "
Jun 15 06:18:09.490435 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:09.490297    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2pcsn\" (UniqueName: \"kubernetes.io/projected/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-kube-api-access-2pcsn\") pod \"7d0e2456-d3c7-4776-a6b8-644cff2e21e1\" (UID: \"7d0e2456-d3c7-4776-a6b8-644cff2e21e1\") "
Jun 15 06:18:09.490435 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:09.490353    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-kserve-provision-location\") pod \"7d0e2456-d3c7-4776-a6b8-644cff2e21e1\" (UID: \"7d0e2456-d3c7-4776-a6b8-644cff2e21e1\") "
Jun 15 06:18:09.490676 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:09.490639    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7d0e2456-d3c7-4776-a6b8-644cff2e21e1" (UID: "7d0e2456-d3c7-4776-a6b8-644cff2e21e1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:18:09.490795 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:09.490683    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7d0e2456-d3c7-4776-a6b8-644cff2e21e1" (UID: "7d0e2456-d3c7-4776-a6b8-644cff2e21e1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:18:09.490914 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:09.490889    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7d0e2456-d3c7-4776-a6b8-644cff2e21e1" (UID: "7d0e2456-d3c7-4776-a6b8-644cff2e21e1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:18:09.492455 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:09.492430    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7d0e2456-d3c7-4776-a6b8-644cff2e21e1" (UID: "7d0e2456-d3c7-4776-a6b8-644cff2e21e1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:18:09.492531 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:09.492491    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-kube-api-access-2pcsn" (OuterVolumeSpecName: "kube-api-access-2pcsn") pod "7d0e2456-d3c7-4776-a6b8-644cff2e21e1" (UID: "7d0e2456-d3c7-4776-a6b8-644cff2e21e1"). InnerVolumeSpecName "kube-api-access-2pcsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:18:09.591516 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:09.591488    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:18:09.591516 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:09.591513    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:18:09.591722 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:09.591524    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:18:09.591722 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:09.591534    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:18:09.591722 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:09.591542    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pcsn\" (UniqueName: \"kubernetes.io/projected/7d0e2456-d3c7-4776-a6b8-644cff2e21e1-kube-api-access-2pcsn\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:18:10.036529 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:10.036496    2572 generic.go:356] "Generic (PLEG): container finished" podID="7d0e2456-d3c7-4776-a6b8-644cff2e21e1" containerID="e13176016adbe81f06bcbf08a3f0e7348b531ef120feb32249f0b27196bcf035" exitCode=0
Jun 15 06:18:10.036943 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:10.036533    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h" event={"ID":"7d0e2456-d3c7-4776-a6b8-644cff2e21e1","Type":"ContainerDied","Data":"e13176016adbe81f06bcbf08a3f0e7348b531ef120feb32249f0b27196bcf035"}
Jun 15 06:18:10.036943 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:10.036552    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"
Jun 15 06:18:10.036943 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:10.036582    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h" event={"ID":"7d0e2456-d3c7-4776-a6b8-644cff2e21e1","Type":"ContainerDied","Data":"263b20048985d22b1e81f0a586d0bec88821a076b0b55ee6a5fcbee58ad64ca6"}
Jun 15 06:18:10.036943 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:10.036609    2572 scope.go:117] "RemoveContainer" containerID="830e9a1660128af09ae3420736b23d00d34863658154503e1cfb2c42e22a4451"
Jun 15 06:18:10.045509 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:10.045490    2572 scope.go:117] "RemoveContainer" containerID="e13176016adbe81f06bcbf08a3f0e7348b531ef120feb32249f0b27196bcf035"
Jun 15 06:18:10.053621 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:10.053602    2572 scope.go:117] "RemoveContainer" containerID="25d4abac1b52e9b7404b595ffc8dae8e6154be64b64de82613e44e004dd23b5d"
Jun 15 06:18:10.059185 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:10.059159    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"]
Jun 15 06:18:10.061156 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:10.061135    2572 scope.go:117] "RemoveContainer" containerID="830e9a1660128af09ae3420736b23d00d34863658154503e1cfb2c42e22a4451"
Jun 15 06:18:10.061558 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:18:10.061515    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"830e9a1660128af09ae3420736b23d00d34863658154503e1cfb2c42e22a4451\": container with ID starting with 830e9a1660128af09ae3420736b23d00d34863658154503e1cfb2c42e22a4451 not found: ID does not exist" containerID="830e9a1660128af09ae3420736b23d00d34863658154503e1cfb2c42e22a4451"
Jun 15 06:18:10.061657 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:10.061557    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"830e9a1660128af09ae3420736b23d00d34863658154503e1cfb2c42e22a4451"} err="failed to get container status \"830e9a1660128af09ae3420736b23d00d34863658154503e1cfb2c42e22a4451\": rpc error: code = NotFound desc = could not find container \"830e9a1660128af09ae3420736b23d00d34863658154503e1cfb2c42e22a4451\": container with ID starting with 830e9a1660128af09ae3420736b23d00d34863658154503e1cfb2c42e22a4451 not found: ID does not exist"
Jun 15 06:18:10.061657 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:10.061582    2572 scope.go:117] "RemoveContainer" containerID="e13176016adbe81f06bcbf08a3f0e7348b531ef120feb32249f0b27196bcf035"
Jun 15 06:18:10.061921 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:18:10.061894    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e13176016adbe81f06bcbf08a3f0e7348b531ef120feb32249f0b27196bcf035\": container with ID starting with e13176016adbe81f06bcbf08a3f0e7348b531ef120feb32249f0b27196bcf035 not found: ID does not exist" containerID="e13176016adbe81f06bcbf08a3f0e7348b531ef120feb32249f0b27196bcf035"
Jun 15 06:18:10.062046 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:10.061930    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e13176016adbe81f06bcbf08a3f0e7348b531ef120feb32249f0b27196bcf035"} err="failed to get container status \"e13176016adbe81f06bcbf08a3f0e7348b531ef120feb32249f0b27196bcf035\": rpc error: code = NotFound desc = could not find container \"e13176016adbe81f06bcbf08a3f0e7348b531ef120feb32249f0b27196bcf035\": container with ID starting with e13176016adbe81f06bcbf08a3f0e7348b531ef120feb32249f0b27196bcf035 not found: ID does not exist"
Jun 15 06:18:10.062046 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:10.061952    2572 scope.go:117] "RemoveContainer" containerID="25d4abac1b52e9b7404b595ffc8dae8e6154be64b64de82613e44e004dd23b5d"
Jun 15 06:18:10.062271 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:18:10.062242    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25d4abac1b52e9b7404b595ffc8dae8e6154be64b64de82613e44e004dd23b5d\": container with ID starting with 25d4abac1b52e9b7404b595ffc8dae8e6154be64b64de82613e44e004dd23b5d not found: ID does not exist" containerID="25d4abac1b52e9b7404b595ffc8dae8e6154be64b64de82613e44e004dd23b5d"
Jun 15 06:18:10.062345 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:10.062270    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25d4abac1b52e9b7404b595ffc8dae8e6154be64b64de82613e44e004dd23b5d"} err="failed to get container status \"25d4abac1b52e9b7404b595ffc8dae8e6154be64b64de82613e44e004dd23b5d\": rpc error: code = NotFound desc = could not find container \"25d4abac1b52e9b7404b595ffc8dae8e6154be64b64de82613e44e004dd23b5d\": container with ID starting with 25d4abac1b52e9b7404b595ffc8dae8e6154be64b64de82613e44e004dd23b5d not found: ID does not exist"
Jun 15 06:18:10.063844 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:10.063826    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-4wb7h"]
Jun 15 06:18:11.043115 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:11.043082    2572 generic.go:356] "Generic (PLEG): container finished" podID="b21d1f4a-8383-425f-a3c1-96c9e65dc9df" containerID="9e25d8b9946ddcde8804e65781fba77f1e38b014735d16ac507b1d8c70753e2c" exitCode=0
Jun 15 06:18:11.043527 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:11.043159    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn" event={"ID":"b21d1f4a-8383-425f-a3c1-96c9e65dc9df","Type":"ContainerDied","Data":"9e25d8b9946ddcde8804e65781fba77f1e38b014735d16ac507b1d8c70753e2c"}
Jun 15 06:18:11.323832 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:11.323750    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d0e2456-d3c7-4776-a6b8-644cff2e21e1" path="/var/lib/kubelet/pods/7d0e2456-d3c7-4776-a6b8-644cff2e21e1/volumes"
Jun 15 06:18:12.049681 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:12.049652    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn" event={"ID":"b21d1f4a-8383-425f-a3c1-96c9e65dc9df","Type":"ContainerStarted","Data":"64b520e0f38fe63a9df41944a8fd64e867ddf50bfd21b0740cb8ef5f45d41a89"}
Jun 15 06:18:12.050048 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:12.049706    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn" event={"ID":"b21d1f4a-8383-425f-a3c1-96c9e65dc9df","Type":"ContainerStarted","Data":"28ed9b787081a87f2e75a7e1624ab697eddc33688e2de3b3559c7a5b17064e61"}
Jun 15 06:18:12.050048 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:12.049892    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn"
Jun 15 06:18:12.077601 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:12.077550    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn" podStartSLOduration=3.183132307 podStartE2EDuration="7.077535098s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.894402791s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:18:12.075040246 +0000 UTC m=+1807.223220243" watchObservedRunningTime="2026-06-15 06:18:12.077535098 +0000 UTC m=+1807.225715095"
Jun 15 06:18:13.054576 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:13.054545    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn"
Jun 15 06:18:13.055803 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:13.055768    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn" podUID="b21d1f4a-8383-425f-a3c1-96c9e65dc9df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jun 15 06:18:14.059058 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:14.059015    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn" podUID="b21d1f4a-8383-425f-a3c1-96c9e65dc9df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jun 15 06:18:19.063249 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:19.063216    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn"
Jun 15 06:18:19.063909 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:19.063876    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn" podUID="b21d1f4a-8383-425f-a3c1-96c9e65dc9df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jun 15 06:18:29.063873 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:29.063834    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn" podUID="b21d1f4a-8383-425f-a3c1-96c9e65dc9df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jun 15 06:18:39.063922 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:39.063885    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn" podUID="b21d1f4a-8383-425f-a3c1-96c9e65dc9df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jun 15 06:18:49.064795 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:49.064751    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn" podUID="b21d1f4a-8383-425f-a3c1-96c9e65dc9df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jun 15 06:18:59.063912 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:18:59.063866    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn" podUID="b21d1f4a-8383-425f-a3c1-96c9e65dc9df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jun 15 06:19:09.064530 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:09.064490    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn" podUID="b21d1f4a-8383-425f-a3c1-96c9e65dc9df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jun 15 06:19:19.063977 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:19.063894    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn" podUID="b21d1f4a-8383-425f-a3c1-96c9e65dc9df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jun 15 06:19:29.064262 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:29.064223    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn" podUID="b21d1f4a-8383-425f-a3c1-96c9e65dc9df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jun 15 06:19:39.065150 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:39.065120    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn"
Jun 15 06:19:47.396970 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.396937    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn"]
Jun 15 06:19:47.397465 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.397179    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn" podUID="b21d1f4a-8383-425f-a3c1-96c9e65dc9df" containerName="kserve-container" containerID="cri-o://28ed9b787081a87f2e75a7e1624ab697eddc33688e2de3b3559c7a5b17064e61" gracePeriod=30
Jun 15 06:19:47.397465 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.397267    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn" podUID="b21d1f4a-8383-425f-a3c1-96c9e65dc9df" containerName="kube-rbac-proxy" containerID="cri-o://64b520e0f38fe63a9df41944a8fd64e867ddf50bfd21b0740cb8ef5f45d41a89" gracePeriod=30
Jun 15 06:19:47.510312 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.510281    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb"]
Jun 15 06:19:47.510714 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.510693    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d0e2456-d3c7-4776-a6b8-644cff2e21e1" containerName="storage-initializer"
Jun 15 06:19:47.510714 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.510709    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d0e2456-d3c7-4776-a6b8-644cff2e21e1" containerName="storage-initializer"
Jun 15 06:19:47.510714 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.510718    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d0e2456-d3c7-4776-a6b8-644cff2e21e1" containerName="kube-rbac-proxy"
Jun 15 06:19:47.510927 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.510725    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d0e2456-d3c7-4776-a6b8-644cff2e21e1" containerName="kube-rbac-proxy"
Jun 15 06:19:47.510927 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.510750    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d0e2456-d3c7-4776-a6b8-644cff2e21e1" containerName="kserve-container"
Jun 15 06:19:47.510927 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.510756    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d0e2456-d3c7-4776-a6b8-644cff2e21e1" containerName="kserve-container"
Jun 15 06:19:47.510927 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.510812    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d0e2456-d3c7-4776-a6b8-644cff2e21e1" containerName="kube-rbac-proxy"
Jun 15 06:19:47.510927 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.510824    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d0e2456-d3c7-4776-a6b8-644cff2e21e1" containerName="kserve-container"
Jun 15 06:19:47.513895 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.513877    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb"
Jun 15 06:19:47.515860 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.515835    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Jun 15 06:19:47.515860 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.515851    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Jun 15 06:19:47.523135 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.523112    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb"]
Jun 15 06:19:47.554169 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.554141    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2a66551-2406-4d19-8359-3853b091cab6-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb\" (UID: \"f2a66551-2406-4d19-8359-3853b091cab6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb"
Jun 15 06:19:47.554323 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.554215    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2a66551-2406-4d19-8359-3853b091cab6-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb\" (UID: \"f2a66551-2406-4d19-8359-3853b091cab6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb"
Jun 15 06:19:47.554323 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.554235    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8hvs\" (UniqueName: \"kubernetes.io/projected/f2a66551-2406-4d19-8359-3853b091cab6-kube-api-access-t8hvs\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb\" (UID: \"f2a66551-2406-4d19-8359-3853b091cab6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb"
Jun 15 06:19:47.554323 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.554254    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2a66551-2406-4d19-8359-3853b091cab6-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb\" (UID: \"f2a66551-2406-4d19-8359-3853b091cab6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb"
Jun 15 06:19:47.554323 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.554304    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2a66551-2406-4d19-8359-3853b091cab6-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb\" (UID: \"f2a66551-2406-4d19-8359-3853b091cab6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb"
Jun 15 06:19:47.655760 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.655662    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2a66551-2406-4d19-8359-3853b091cab6-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb\" (UID: \"f2a66551-2406-4d19-8359-3853b091cab6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb"
Jun 15 06:19:47.655760 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.655697    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8hvs\" (UniqueName: \"kubernetes.io/projected/f2a66551-2406-4d19-8359-3853b091cab6-kube-api-access-t8hvs\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb\" (UID: \"f2a66551-2406-4d19-8359-3853b091cab6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb"
Jun 15 06:19:47.655760 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.655714    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2a66551-2406-4d19-8359-3853b091cab6-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb\" (UID: \"f2a66551-2406-4d19-8359-3853b091cab6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb"
Jun 15 06:19:47.655760 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.655736    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2a66551-2406-4d19-8359-3853b091cab6-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb\" (UID: \"f2a66551-2406-4d19-8359-3853b091cab6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb"
Jun 15 06:19:47.656114 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.655804    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2a66551-2406-4d19-8359-3853b091cab6-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb\" (UID: \"f2a66551-2406-4d19-8359-3853b091cab6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb"
Jun 15 06:19:47.656114 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.656074    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2a66551-2406-4d19-8359-3853b091cab6-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb\" (UID: \"f2a66551-2406-4d19-8359-3853b091cab6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb"
Jun 15 06:19:47.656468 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.656438    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2a66551-2406-4d19-8359-3853b091cab6-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb\" (UID: \"f2a66551-2406-4d19-8359-3853b091cab6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb"
Jun 15 06:19:47.656530 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.656449    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2a66551-2406-4d19-8359-3853b091cab6-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb\" (UID: \"f2a66551-2406-4d19-8359-3853b091cab6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb"
Jun 15 06:19:47.658243 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.658222    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2a66551-2406-4d19-8359-3853b091cab6-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb\" (UID: \"f2a66551-2406-4d19-8359-3853b091cab6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb"
Jun 15 06:19:47.667490 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.667460    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8hvs\" (UniqueName: \"kubernetes.io/projected/f2a66551-2406-4d19-8359-3853b091cab6-kube-api-access-t8hvs\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb\" (UID: \"f2a66551-2406-4d19-8359-3853b091cab6\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb"
Jun 15 06:19:47.825070 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.825034    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb"
Jun 15 06:19:47.955776 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:47.955691    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb"]
Jun 15 06:19:47.958473 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:19:47.958448    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2a66551_2406_4d19_8359_3853b091cab6.slice/crio-b4e04e088e6e1a474033ebf26605c85ad6feff6cad448566e0c714e3f1efcab7 WatchSource:0}: Error finding container b4e04e088e6e1a474033ebf26605c85ad6feff6cad448566e0c714e3f1efcab7: Status 404 returned error can't find the container with id b4e04e088e6e1a474033ebf26605c85ad6feff6cad448566e0c714e3f1efcab7
Jun 15 06:19:48.499627 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:48.499589    2572 generic.go:356] "Generic (PLEG): container finished" podID="b21d1f4a-8383-425f-a3c1-96c9e65dc9df" containerID="64b520e0f38fe63a9df41944a8fd64e867ddf50bfd21b0740cb8ef5f45d41a89" exitCode=2
Jun 15 06:19:48.499998 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:48.499657    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn" event={"ID":"b21d1f4a-8383-425f-a3c1-96c9e65dc9df","Type":"ContainerDied","Data":"64b520e0f38fe63a9df41944a8fd64e867ddf50bfd21b0740cb8ef5f45d41a89"}
Jun 15 06:19:48.501116 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:48.501094    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb" event={"ID":"f2a66551-2406-4d19-8359-3853b091cab6","Type":"ContainerStarted","Data":"d08b365870bef7f621351b699281da1a71881923a54b2575114469720e4dde6b"}
Jun 15 06:19:48.501189 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:48.501147    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb" event={"ID":"f2a66551-2406-4d19-8359-3853b091cab6","Type":"ContainerStarted","Data":"b4e04e088e6e1a474033ebf26605c85ad6feff6cad448566e0c714e3f1efcab7"}
Jun 15 06:19:49.059540 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:49.059497    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn" podUID="b21d1f4a-8383-425f-a3c1-96c9e65dc9df" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Jun 15 06:19:49.063821 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:49.063793    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn" podUID="b21d1f4a-8383-425f-a3c1-96c9e65dc9df" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jun 15 06:19:51.042992 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.042970    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn"
Jun 15 06:19:51.087064 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.087037    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-kserve-provision-location\") pod \"b21d1f4a-8383-425f-a3c1-96c9e65dc9df\" (UID: \"b21d1f4a-8383-425f-a3c1-96c9e65dc9df\") "
Jun 15 06:19:51.087226 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.087080    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-proxy-tls\") pod \"b21d1f4a-8383-425f-a3c1-96c9e65dc9df\" (UID: \"b21d1f4a-8383-425f-a3c1-96c9e65dc9df\") "
Jun 15 06:19:51.087226 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.087122    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-kube-rbac-proxy-sar-config\") pod \"b21d1f4a-8383-425f-a3c1-96c9e65dc9df\" (UID: \"b21d1f4a-8383-425f-a3c1-96c9e65dc9df\") "
Jun 15 06:19:51.087226 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.087143    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxthn\" (UniqueName: \"kubernetes.io/projected/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-kube-api-access-wxthn\") pod \"b21d1f4a-8383-425f-a3c1-96c9e65dc9df\" (UID: \"b21d1f4a-8383-425f-a3c1-96c9e65dc9df\") "
Jun 15 06:19:51.087426 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.087238    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-cabundle-cert\") pod \"b21d1f4a-8383-425f-a3c1-96c9e65dc9df\" (UID: \"b21d1f4a-8383-425f-a3c1-96c9e65dc9df\") "
Jun 15 06:19:51.087426 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.087337    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b21d1f4a-8383-425f-a3c1-96c9e65dc9df" (UID: "b21d1f4a-8383-425f-a3c1-96c9e65dc9df"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:19:51.087630 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.087505    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b21d1f4a-8383-425f-a3c1-96c9e65dc9df" (UID: "b21d1f4a-8383-425f-a3c1-96c9e65dc9df"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:19:51.087743 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.087559    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:19:51.087743 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.087663    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b21d1f4a-8383-425f-a3c1-96c9e65dc9df" (UID: "b21d1f4a-8383-425f-a3c1-96c9e65dc9df"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:19:51.089230 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.089178    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b21d1f4a-8383-425f-a3c1-96c9e65dc9df" (UID: "b21d1f4a-8383-425f-a3c1-96c9e65dc9df"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:19:51.089315 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.089260    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-kube-api-access-wxthn" (OuterVolumeSpecName: "kube-api-access-wxthn") pod "b21d1f4a-8383-425f-a3c1-96c9e65dc9df" (UID: "b21d1f4a-8383-425f-a3c1-96c9e65dc9df"). InnerVolumeSpecName "kube-api-access-wxthn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:19:51.188744 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.188710    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:19:51.188744 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.188742    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:19:51.188744 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.188751    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:19:51.188977 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.188761    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxthn\" (UniqueName: \"kubernetes.io/projected/b21d1f4a-8383-425f-a3c1-96c9e65dc9df-kube-api-access-wxthn\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:19:51.517313 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.517275    2572 generic.go:356] "Generic (PLEG): container finished" podID="b21d1f4a-8383-425f-a3c1-96c9e65dc9df" containerID="28ed9b787081a87f2e75a7e1624ab697eddc33688e2de3b3559c7a5b17064e61" exitCode=0
Jun 15 06:19:51.517499 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.517340    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn"
Jun 15 06:19:51.517499 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.517349    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn" event={"ID":"b21d1f4a-8383-425f-a3c1-96c9e65dc9df","Type":"ContainerDied","Data":"28ed9b787081a87f2e75a7e1624ab697eddc33688e2de3b3559c7a5b17064e61"}
Jun 15 06:19:51.517499 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.517427    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn" event={"ID":"b21d1f4a-8383-425f-a3c1-96c9e65dc9df","Type":"ContainerDied","Data":"4aba301478ec7ac193ead881a984cf0de85d1879744c0c7bb67ce044cf4d1855"}
Jun 15 06:19:51.517499 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.517462    2572 scope.go:117] "RemoveContainer" containerID="64b520e0f38fe63a9df41944a8fd64e867ddf50bfd21b0740cb8ef5f45d41a89"
Jun 15 06:19:51.525804 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.525783    2572 scope.go:117] "RemoveContainer" containerID="28ed9b787081a87f2e75a7e1624ab697eddc33688e2de3b3559c7a5b17064e61"
Jun 15 06:19:51.533132 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.533114    2572 scope.go:117] "RemoveContainer" containerID="9e25d8b9946ddcde8804e65781fba77f1e38b014735d16ac507b1d8c70753e2c"
Jun 15 06:19:51.538022 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.537998    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn"]
Jun 15 06:19:51.541339 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.541319    2572 scope.go:117] "RemoveContainer" containerID="64b520e0f38fe63a9df41944a8fd64e867ddf50bfd21b0740cb8ef5f45d41a89"
Jun 15 06:19:51.541666 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:19:51.541646    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64b520e0f38fe63a9df41944a8fd64e867ddf50bfd21b0740cb8ef5f45d41a89\": container with ID starting with 64b520e0f38fe63a9df41944a8fd64e867ddf50bfd21b0740cb8ef5f45d41a89 not found: ID does not exist" containerID="64b520e0f38fe63a9df41944a8fd64e867ddf50bfd21b0740cb8ef5f45d41a89"
Jun 15 06:19:51.541732 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.541675    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64b520e0f38fe63a9df41944a8fd64e867ddf50bfd21b0740cb8ef5f45d41a89"} err="failed to get container status \"64b520e0f38fe63a9df41944a8fd64e867ddf50bfd21b0740cb8ef5f45d41a89\": rpc error: code = NotFound desc = could not find container \"64b520e0f38fe63a9df41944a8fd64e867ddf50bfd21b0740cb8ef5f45d41a89\": container with ID starting with 64b520e0f38fe63a9df41944a8fd64e867ddf50bfd21b0740cb8ef5f45d41a89 not found: ID does not exist"
Jun 15 06:19:51.541732 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.541694    2572 scope.go:117] "RemoveContainer" containerID="28ed9b787081a87f2e75a7e1624ab697eddc33688e2de3b3559c7a5b17064e61"
Jun 15 06:19:51.541940 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:19:51.541919    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28ed9b787081a87f2e75a7e1624ab697eddc33688e2de3b3559c7a5b17064e61\": container with ID starting with 28ed9b787081a87f2e75a7e1624ab697eddc33688e2de3b3559c7a5b17064e61 not found: ID does not exist" containerID="28ed9b787081a87f2e75a7e1624ab697eddc33688e2de3b3559c7a5b17064e61"
Jun 15 06:19:51.541983 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.541947    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28ed9b787081a87f2e75a7e1624ab697eddc33688e2de3b3559c7a5b17064e61"} err="failed to get container status \"28ed9b787081a87f2e75a7e1624ab697eddc33688e2de3b3559c7a5b17064e61\": rpc error: code = NotFound desc = could not find container \"28ed9b787081a87f2e75a7e1624ab697eddc33688e2de3b3559c7a5b17064e61\": container with ID starting with 28ed9b787081a87f2e75a7e1624ab697eddc33688e2de3b3559c7a5b17064e61 not found: ID does not exist"
Jun 15 06:19:51.541983 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.541963    2572 scope.go:117] "RemoveContainer" containerID="9e25d8b9946ddcde8804e65781fba77f1e38b014735d16ac507b1d8c70753e2c"
Jun 15 06:19:51.542230 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:19:51.542213    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e25d8b9946ddcde8804e65781fba77f1e38b014735d16ac507b1d8c70753e2c\": container with ID starting with 9e25d8b9946ddcde8804e65781fba77f1e38b014735d16ac507b1d8c70753e2c not found: ID does not exist" containerID="9e25d8b9946ddcde8804e65781fba77f1e38b014735d16ac507b1d8c70753e2c"
Jun 15 06:19:51.542274 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.542235    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e25d8b9946ddcde8804e65781fba77f1e38b014735d16ac507b1d8c70753e2c"} err="failed to get container status \"9e25d8b9946ddcde8804e65781fba77f1e38b014735d16ac507b1d8c70753e2c\": rpc error: code = NotFound desc = could not find container \"9e25d8b9946ddcde8804e65781fba77f1e38b014735d16ac507b1d8c70753e2c\": container with ID starting with 9e25d8b9946ddcde8804e65781fba77f1e38b014735d16ac507b1d8c70753e2c not found: ID does not exist"
Jun 15 06:19:51.542666 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:51.542649    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-runtime-predictor-885fc8777-2xqrn"]
Jun 15 06:19:52.523589 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:52.523556    2572 generic.go:356] "Generic (PLEG): container finished" podID="f2a66551-2406-4d19-8359-3853b091cab6" containerID="d08b365870bef7f621351b699281da1a71881923a54b2575114469720e4dde6b" exitCode=0
Jun 15 06:19:52.524005 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:52.523628    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb" event={"ID":"f2a66551-2406-4d19-8359-3853b091cab6","Type":"ContainerDied","Data":"d08b365870bef7f621351b699281da1a71881923a54b2575114469720e4dde6b"}
Jun 15 06:19:53.324107 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:53.324075    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b21d1f4a-8383-425f-a3c1-96c9e65dc9df" path="/var/lib/kubelet/pods/b21d1f4a-8383-425f-a3c1-96c9e65dc9df/volumes"
Jun 15 06:19:53.533277 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:53.533241    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb" event={"ID":"f2a66551-2406-4d19-8359-3853b091cab6","Type":"ContainerStarted","Data":"dbd51a38f3e3151766b088674543b6e5e2657360f079c96ccc020fe1bcfe0c94"}
Jun 15 06:19:53.533671 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:53.533312    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb" event={"ID":"f2a66551-2406-4d19-8359-3853b091cab6","Type":"ContainerStarted","Data":"581e90e0abcaa033019cd222eecf8e1d7a0c6048c7f269e2a6f8bbcb15df59cd"}
Jun 15 06:19:53.533671 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:53.533527    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb"
Jun 15 06:19:53.553036 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:53.552990    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb" podStartSLOduration=2.766554815 podStartE2EDuration="6.55297605s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.786421235s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:19:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:19:53.551788396 +0000 UTC m=+1908.699968394" watchObservedRunningTime="2026-06-15 06:19:53.55297605 +0000 UTC m=+1908.701156046"
Jun 15 06:19:54.538928 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:54.538894    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb"
Jun 15 06:19:54.540177 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:54.540150    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb" podUID="f2a66551-2406-4d19-8359-3853b091cab6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jun 15 06:19:55.544201 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:19:55.544157    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb" podUID="f2a66551-2406-4d19-8359-3853b091cab6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jun 15 06:20:00.549420 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:20:00.549383    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb"
Jun 15 06:20:00.549926 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:20:00.549897    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb" podUID="f2a66551-2406-4d19-8359-3853b091cab6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jun 15 06:20:10.549954 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:20:10.549917    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb" podUID="f2a66551-2406-4d19-8359-3853b091cab6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jun 15 06:20:20.550561 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:20:20.550523    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb" podUID="f2a66551-2406-4d19-8359-3853b091cab6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jun 15 06:20:30.550329 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:20:30.550288    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb" podUID="f2a66551-2406-4d19-8359-3853b091cab6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jun 15 06:20:40.550732 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:20:40.550691    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb" podUID="f2a66551-2406-4d19-8359-3853b091cab6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jun 15 06:20:50.549873 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:20:50.549792    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb" podUID="f2a66551-2406-4d19-8359-3853b091cab6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jun 15 06:21:00.550453 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:00.550413    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb" podUID="f2a66551-2406-4d19-8359-3853b091cab6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jun 15 06:21:02.319648 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:02.319601    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb" podUID="f2a66551-2406-4d19-8359-3853b091cab6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jun 15 06:21:12.320267 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:12.320234    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb"
Jun 15 06:21:18.521228 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.521192    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb"]
Jun 15 06:21:18.521683 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.521447    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb" podUID="f2a66551-2406-4d19-8359-3853b091cab6" containerName="kserve-container" containerID="cri-o://581e90e0abcaa033019cd222eecf8e1d7a0c6048c7f269e2a6f8bbcb15df59cd" gracePeriod=30
Jun 15 06:21:18.521683 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.521482    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb" podUID="f2a66551-2406-4d19-8359-3853b091cab6" containerName="kube-rbac-proxy" containerID="cri-o://dbd51a38f3e3151766b088674543b6e5e2657360f079c96ccc020fe1bcfe0c94" gracePeriod=30
Jun 15 06:21:18.653041 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.653010    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq"]
Jun 15 06:21:18.653390 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.653377    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b21d1f4a-8383-425f-a3c1-96c9e65dc9df" containerName="kube-rbac-proxy"
Jun 15 06:21:18.653445 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.653391    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b21d1f4a-8383-425f-a3c1-96c9e65dc9df" containerName="kube-rbac-proxy"
Jun 15 06:21:18.653445 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.653410    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b21d1f4a-8383-425f-a3c1-96c9e65dc9df" containerName="storage-initializer"
Jun 15 06:21:18.653445 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.653416    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b21d1f4a-8383-425f-a3c1-96c9e65dc9df" containerName="storage-initializer"
Jun 15 06:21:18.653445 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.653433    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b21d1f4a-8383-425f-a3c1-96c9e65dc9df" containerName="kserve-container"
Jun 15 06:21:18.653445 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.653439    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="b21d1f4a-8383-425f-a3c1-96c9e65dc9df" containerName="kserve-container"
Jun 15 06:21:18.653601 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.653490    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b21d1f4a-8383-425f-a3c1-96c9e65dc9df" containerName="kserve-container"
Jun 15 06:21:18.653601 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.653500    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="b21d1f4a-8383-425f-a3c1-96c9e65dc9df" containerName="kube-rbac-proxy"
Jun 15 06:21:18.656439 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.656424    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq"
Jun 15 06:21:18.658325 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.658302    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-452881-predictor-serving-cert\""
Jun 15 06:21:18.658555 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.658536    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-452881-kube-rbac-proxy-sar-config\""
Jun 15 06:21:18.672934 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.672911    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq"]
Jun 15 06:21:18.786051 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.785943    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rst96\" (UniqueName: \"kubernetes.io/projected/f16264e1-fa3f-43e4-9909-ef67243c1c71-kube-api-access-rst96\") pod \"isvc-primary-452881-predictor-6bd5b98f4d-sskgq\" (UID: \"f16264e1-fa3f-43e4-9909-ef67243c1c71\") " pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq"
Jun 15 06:21:18.786051 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.785998    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f16264e1-fa3f-43e4-9909-ef67243c1c71-proxy-tls\") pod \"isvc-primary-452881-predictor-6bd5b98f4d-sskgq\" (UID: \"f16264e1-fa3f-43e4-9909-ef67243c1c71\") " pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq"
Jun 15 06:21:18.786051 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.786029    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f16264e1-fa3f-43e4-9909-ef67243c1c71-kserve-provision-location\") pod \"isvc-primary-452881-predictor-6bd5b98f4d-sskgq\" (UID: \"f16264e1-fa3f-43e4-9909-ef67243c1c71\") " pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq"
Jun 15 06:21:18.786298 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.786149    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f16264e1-fa3f-43e4-9909-ef67243c1c71-kube-rbac-proxy-sar-config\") pod \"isvc-primary-452881-predictor-6bd5b98f4d-sskgq\" (UID: \"f16264e1-fa3f-43e4-9909-ef67243c1c71\") " pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq"
Jun 15 06:21:18.786298 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.786174    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f16264e1-fa3f-43e4-9909-ef67243c1c71-cabundle-cert\") pod \"isvc-primary-452881-predictor-6bd5b98f4d-sskgq\" (UID: \"f16264e1-fa3f-43e4-9909-ef67243c1c71\") " pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq"
Jun 15 06:21:18.886751 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.886716    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f16264e1-fa3f-43e4-9909-ef67243c1c71-kube-rbac-proxy-sar-config\") pod \"isvc-primary-452881-predictor-6bd5b98f4d-sskgq\" (UID: \"f16264e1-fa3f-43e4-9909-ef67243c1c71\") " pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq"
Jun 15 06:21:18.886751 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.886755    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f16264e1-fa3f-43e4-9909-ef67243c1c71-cabundle-cert\") pod \"isvc-primary-452881-predictor-6bd5b98f4d-sskgq\" (UID: \"f16264e1-fa3f-43e4-9909-ef67243c1c71\") " pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq"
Jun 15 06:21:18.886990 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.886793    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rst96\" (UniqueName: \"kubernetes.io/projected/f16264e1-fa3f-43e4-9909-ef67243c1c71-kube-api-access-rst96\") pod \"isvc-primary-452881-predictor-6bd5b98f4d-sskgq\" (UID: \"f16264e1-fa3f-43e4-9909-ef67243c1c71\") " pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq"
Jun 15 06:21:18.886990 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.886815    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f16264e1-fa3f-43e4-9909-ef67243c1c71-proxy-tls\") pod \"isvc-primary-452881-predictor-6bd5b98f4d-sskgq\" (UID: \"f16264e1-fa3f-43e4-9909-ef67243c1c71\") " pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq"
Jun 15 06:21:18.886990 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.886841    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f16264e1-fa3f-43e4-9909-ef67243c1c71-kserve-provision-location\") pod \"isvc-primary-452881-predictor-6bd5b98f4d-sskgq\" (UID: \"f16264e1-fa3f-43e4-9909-ef67243c1c71\") " pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq"
Jun 15 06:21:18.887254 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.887233    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f16264e1-fa3f-43e4-9909-ef67243c1c71-kserve-provision-location\") pod \"isvc-primary-452881-predictor-6bd5b98f4d-sskgq\" (UID: \"f16264e1-fa3f-43e4-9909-ef67243c1c71\") " pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq"
Jun 15 06:21:18.887563 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.887541    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f16264e1-fa3f-43e4-9909-ef67243c1c71-kube-rbac-proxy-sar-config\") pod \"isvc-primary-452881-predictor-6bd5b98f4d-sskgq\" (UID: \"f16264e1-fa3f-43e4-9909-ef67243c1c71\") " pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq"
Jun 15 06:21:18.887610 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.887541    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f16264e1-fa3f-43e4-9909-ef67243c1c71-cabundle-cert\") pod \"isvc-primary-452881-predictor-6bd5b98f4d-sskgq\" (UID: \"f16264e1-fa3f-43e4-9909-ef67243c1c71\") " pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq"
Jun 15 06:21:18.889327 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.889311    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f16264e1-fa3f-43e4-9909-ef67243c1c71-proxy-tls\") pod \"isvc-primary-452881-predictor-6bd5b98f4d-sskgq\" (UID: \"f16264e1-fa3f-43e4-9909-ef67243c1c71\") " pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq"
Jun 15 06:21:18.896220 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.896199    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rst96\" (UniqueName: \"kubernetes.io/projected/f16264e1-fa3f-43e4-9909-ef67243c1c71-kube-api-access-rst96\") pod \"isvc-primary-452881-predictor-6bd5b98f4d-sskgq\" (UID: \"f16264e1-fa3f-43e4-9909-ef67243c1c71\") " pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq"
Jun 15 06:21:18.931905 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.931875    2572 generic.go:356] "Generic (PLEG): container finished" podID="f2a66551-2406-4d19-8359-3853b091cab6" containerID="dbd51a38f3e3151766b088674543b6e5e2657360f079c96ccc020fe1bcfe0c94" exitCode=2
Jun 15 06:21:18.932064 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.931912    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb" event={"ID":"f2a66551-2406-4d19-8359-3853b091cab6","Type":"ContainerDied","Data":"dbd51a38f3e3151766b088674543b6e5e2657360f079c96ccc020fe1bcfe0c94"}
Jun 15 06:21:18.967039 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:18.967006    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq"
Jun 15 06:21:19.101042 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:19.101016    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq"]
Jun 15 06:21:19.102602 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:21:19.102578    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf16264e1_fa3f_43e4_9909_ef67243c1c71.slice/crio-54cfbe79d2ba299396475d86f102588b2802dc503037cd9e46360491eb7eb842 WatchSource:0}: Error finding container 54cfbe79d2ba299396475d86f102588b2802dc503037cd9e46360491eb7eb842: Status 404 returned error can't find the container with id 54cfbe79d2ba299396475d86f102588b2802dc503037cd9e46360491eb7eb842
Jun 15 06:21:19.938374 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:19.938323    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq" event={"ID":"f16264e1-fa3f-43e4-9909-ef67243c1c71","Type":"ContainerStarted","Data":"5c187434aa2af003850ad3fb80f4944b4477cc75987c1cd025a1dfe2a8d49a63"}
Jun 15 06:21:19.938789 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:19.938401    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq" event={"ID":"f16264e1-fa3f-43e4-9909-ef67243c1c71","Type":"ContainerStarted","Data":"54cfbe79d2ba299396475d86f102588b2802dc503037cd9e46360491eb7eb842"}
Jun 15 06:21:20.544692 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:20.544654    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb" podUID="f2a66551-2406-4d19-8359-3853b091cab6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Jun 15 06:21:22.166095 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.166070    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb"
Jun 15 06:21:22.316203 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.316106    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8hvs\" (UniqueName: \"kubernetes.io/projected/f2a66551-2406-4d19-8359-3853b091cab6-kube-api-access-t8hvs\") pod \"f2a66551-2406-4d19-8359-3853b091cab6\" (UID: \"f2a66551-2406-4d19-8359-3853b091cab6\") "
Jun 15 06:21:22.316203 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.316147    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2a66551-2406-4d19-8359-3853b091cab6-kserve-provision-location\") pod \"f2a66551-2406-4d19-8359-3853b091cab6\" (UID: \"f2a66551-2406-4d19-8359-3853b091cab6\") "
Jun 15 06:21:22.316203 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.316178    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2a66551-2406-4d19-8359-3853b091cab6-kube-rbac-proxy-sar-config\") pod \"f2a66551-2406-4d19-8359-3853b091cab6\" (UID: \"f2a66551-2406-4d19-8359-3853b091cab6\") "
Jun 15 06:21:22.316203 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.316209    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2a66551-2406-4d19-8359-3853b091cab6-proxy-tls\") pod \"f2a66551-2406-4d19-8359-3853b091cab6\" (UID: \"f2a66551-2406-4d19-8359-3853b091cab6\") "
Jun 15 06:21:22.316591 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.316458    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2a66551-2406-4d19-8359-3853b091cab6-cabundle-cert\") pod \"f2a66551-2406-4d19-8359-3853b091cab6\" (UID: \"f2a66551-2406-4d19-8359-3853b091cab6\") "
Jun 15 06:21:22.316591 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.316511    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2a66551-2406-4d19-8359-3853b091cab6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f2a66551-2406-4d19-8359-3853b091cab6" (UID: "f2a66551-2406-4d19-8359-3853b091cab6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:21:22.316691 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.316598    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2a66551-2406-4d19-8359-3853b091cab6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f2a66551-2406-4d19-8359-3853b091cab6" (UID: "f2a66551-2406-4d19-8359-3853b091cab6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:21:22.316771 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.316750    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2a66551-2406-4d19-8359-3853b091cab6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f2a66551-2406-4d19-8359-3853b091cab6" (UID: "f2a66551-2406-4d19-8359-3853b091cab6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:21:22.316850 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.316829    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f2a66551-2406-4d19-8359-3853b091cab6-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:21:22.316850 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.316848    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f2a66551-2406-4d19-8359-3853b091cab6-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:21:22.316999 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.316858    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f2a66551-2406-4d19-8359-3853b091cab6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:21:22.318865 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.318844    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2a66551-2406-4d19-8359-3853b091cab6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f2a66551-2406-4d19-8359-3853b091cab6" (UID: "f2a66551-2406-4d19-8359-3853b091cab6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:21:22.318956 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.318860    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2a66551-2406-4d19-8359-3853b091cab6-kube-api-access-t8hvs" (OuterVolumeSpecName: "kube-api-access-t8hvs") pod "f2a66551-2406-4d19-8359-3853b091cab6" (UID: "f2a66551-2406-4d19-8359-3853b091cab6"). InnerVolumeSpecName "kube-api-access-t8hvs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:21:22.418153 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.418105    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8hvs\" (UniqueName: \"kubernetes.io/projected/f2a66551-2406-4d19-8359-3853b091cab6-kube-api-access-t8hvs\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:21:22.418153 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.418148    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f2a66551-2406-4d19-8359-3853b091cab6-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:21:22.954183 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.954117    2572 generic.go:356] "Generic (PLEG): container finished" podID="f2a66551-2406-4d19-8359-3853b091cab6" containerID="581e90e0abcaa033019cd222eecf8e1d7a0c6048c7f269e2a6f8bbcb15df59cd" exitCode=0
Jun 15 06:21:22.954299 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.954178    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb"
Jun 15 06:21:22.954299 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.954198    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb" event={"ID":"f2a66551-2406-4d19-8359-3853b091cab6","Type":"ContainerDied","Data":"581e90e0abcaa033019cd222eecf8e1d7a0c6048c7f269e2a6f8bbcb15df59cd"}
Jun 15 06:21:22.954299 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.954263    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb" event={"ID":"f2a66551-2406-4d19-8359-3853b091cab6","Type":"ContainerDied","Data":"b4e04e088e6e1a474033ebf26605c85ad6feff6cad448566e0c714e3f1efcab7"}
Jun 15 06:21:22.954442 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.954303    2572 scope.go:117] "RemoveContainer" containerID="dbd51a38f3e3151766b088674543b6e5e2657360f079c96ccc020fe1bcfe0c94"
Jun 15 06:21:22.955738 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.955710    2572 generic.go:356] "Generic (PLEG): container finished" podID="f16264e1-fa3f-43e4-9909-ef67243c1c71" containerID="5c187434aa2af003850ad3fb80f4944b4477cc75987c1cd025a1dfe2a8d49a63" exitCode=0
Jun 15 06:21:22.955851 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.955785    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq" event={"ID":"f16264e1-fa3f-43e4-9909-ef67243c1c71","Type":"ContainerDied","Data":"5c187434aa2af003850ad3fb80f4944b4477cc75987c1cd025a1dfe2a8d49a63"}
Jun 15 06:21:22.963827 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.963811    2572 scope.go:117] "RemoveContainer" containerID="581e90e0abcaa033019cd222eecf8e1d7a0c6048c7f269e2a6f8bbcb15df59cd"
Jun 15 06:21:22.972669 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.972653    2572 scope.go:117] "RemoveContainer" containerID="d08b365870bef7f621351b699281da1a71881923a54b2575114469720e4dde6b"
Jun 15 06:21:22.986335 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.986314    2572 scope.go:117] "RemoveContainer" containerID="dbd51a38f3e3151766b088674543b6e5e2657360f079c96ccc020fe1bcfe0c94"
Jun 15 06:21:22.986811 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:21:22.986785    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbd51a38f3e3151766b088674543b6e5e2657360f079c96ccc020fe1bcfe0c94\": container with ID starting with dbd51a38f3e3151766b088674543b6e5e2657360f079c96ccc020fe1bcfe0c94 not found: ID does not exist" containerID="dbd51a38f3e3151766b088674543b6e5e2657360f079c96ccc020fe1bcfe0c94"
Jun 15 06:21:22.986902 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.986821    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbd51a38f3e3151766b088674543b6e5e2657360f079c96ccc020fe1bcfe0c94"} err="failed to get container status \"dbd51a38f3e3151766b088674543b6e5e2657360f079c96ccc020fe1bcfe0c94\": rpc error: code = NotFound desc = could not find container \"dbd51a38f3e3151766b088674543b6e5e2657360f079c96ccc020fe1bcfe0c94\": container with ID starting with dbd51a38f3e3151766b088674543b6e5e2657360f079c96ccc020fe1bcfe0c94 not found: ID does not exist"
Jun 15 06:21:22.986902 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.986844    2572 scope.go:117] "RemoveContainer" containerID="581e90e0abcaa033019cd222eecf8e1d7a0c6048c7f269e2a6f8bbcb15df59cd"
Jun 15 06:21:22.987090 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:21:22.987072    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"581e90e0abcaa033019cd222eecf8e1d7a0c6048c7f269e2a6f8bbcb15df59cd\": container with ID starting with 581e90e0abcaa033019cd222eecf8e1d7a0c6048c7f269e2a6f8bbcb15df59cd not found: ID does not exist" containerID="581e90e0abcaa033019cd222eecf8e1d7a0c6048c7f269e2a6f8bbcb15df59cd"
Jun 15 06:21:22.987132 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.987095    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"581e90e0abcaa033019cd222eecf8e1d7a0c6048c7f269e2a6f8bbcb15df59cd"} err="failed to get container status \"581e90e0abcaa033019cd222eecf8e1d7a0c6048c7f269e2a6f8bbcb15df59cd\": rpc error: code = NotFound desc = could not find container \"581e90e0abcaa033019cd222eecf8e1d7a0c6048c7f269e2a6f8bbcb15df59cd\": container with ID starting with 581e90e0abcaa033019cd222eecf8e1d7a0c6048c7f269e2a6f8bbcb15df59cd not found: ID does not exist"
Jun 15 06:21:22.987132 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.987112    2572 scope.go:117] "RemoveContainer" containerID="d08b365870bef7f621351b699281da1a71881923a54b2575114469720e4dde6b"
Jun 15 06:21:22.987391 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:21:22.987349    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d08b365870bef7f621351b699281da1a71881923a54b2575114469720e4dde6b\": container with ID starting with d08b365870bef7f621351b699281da1a71881923a54b2575114469720e4dde6b not found: ID does not exist" containerID="d08b365870bef7f621351b699281da1a71881923a54b2575114469720e4dde6b"
Jun 15 06:21:22.987498 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:22.987394    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d08b365870bef7f621351b699281da1a71881923a54b2575114469720e4dde6b"} err="failed to get container status \"d08b365870bef7f621351b699281da1a71881923a54b2575114469720e4dde6b\": rpc error: code = NotFound desc = could not find container \"d08b365870bef7f621351b699281da1a71881923a54b2575114469720e4dde6b\": container with ID starting with d08b365870bef7f621351b699281da1a71881923a54b2575114469720e4dde6b not found: ID does not exist"
Jun 15 06:21:23.027149 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:23.027125    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb"]
Jun 15 06:21:23.034036 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:23.034011    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-slmfb"]
Jun 15 06:21:23.324528 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:23.324434    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2a66551-2406-4d19-8359-3853b091cab6" path="/var/lib/kubelet/pods/f2a66551-2406-4d19-8359-3853b091cab6/volumes"
Jun 15 06:21:23.963000 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:23.962955    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq" event={"ID":"f16264e1-fa3f-43e4-9909-ef67243c1c71","Type":"ContainerStarted","Data":"5ffc58de1236cf1c1bf3b998e5825bb92a9ac7c23cdacfaf7eecb0c4e614de5e"}
Jun 15 06:21:23.963162 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:23.963035    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq" event={"ID":"f16264e1-fa3f-43e4-9909-ef67243c1c71","Type":"ContainerStarted","Data":"1fb108b8930eb9035e469819defc8522e74a30bcf73d557b7aaf14a17b7c7711"}
Jun 15 06:21:23.963386 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:23.963345    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq"
Jun 15 06:21:23.988172 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:23.988122    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq" podStartSLOduration=2.445737085 podStartE2EDuration="5.988108623s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.542371538s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:21:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:21:23.986106204 +0000 UTC m=+1999.134286201" watchObservedRunningTime="2026-06-15 06:21:23.988108623 +0000 UTC m=+1999.136288658"
Jun 15 06:21:24.968842 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:24.968807    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq"
Jun 15 06:21:24.970302 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:24.970272    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq" podUID="f16264e1-fa3f-43e4-9909-ef67243c1c71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jun 15 06:21:25.974087 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:25.973799    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq" podUID="f16264e1-fa3f-43e4-9909-ef67243c1c71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jun 15 06:21:30.978802 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:30.978766    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq"
Jun 15 06:21:30.979343 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:30.979313    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq" podUID="f16264e1-fa3f-43e4-9909-ef67243c1c71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jun 15 06:21:40.979539 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:40.979500    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq" podUID="f16264e1-fa3f-43e4-9909-ef67243c1c71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jun 15 06:21:50.979505 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:21:50.979465    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq" podUID="f16264e1-fa3f-43e4-9909-ef67243c1c71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jun 15 06:22:00.979215 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:00.979169    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq" podUID="f16264e1-fa3f-43e4-9909-ef67243c1c71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jun 15 06:22:10.980132 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:10.980090    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq" podUID="f16264e1-fa3f-43e4-9909-ef67243c1c71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jun 15 06:22:20.980096 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:20.980011    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq" podUID="f16264e1-fa3f-43e4-9909-ef67243c1c71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jun 15 06:22:30.979272 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:30.979230    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq" podUID="f16264e1-fa3f-43e4-9909-ef67243c1c71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jun 15 06:22:40.320191 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:40.320160    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq"
Jun 15 06:22:48.808425 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:48.808388    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx"]
Jun 15 06:22:48.808803 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:48.808739    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2a66551-2406-4d19-8359-3853b091cab6" containerName="kserve-container"
Jun 15 06:22:48.808803 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:48.808750    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2a66551-2406-4d19-8359-3853b091cab6" containerName="kserve-container"
Jun 15 06:22:48.808803 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:48.808774    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2a66551-2406-4d19-8359-3853b091cab6" containerName="kube-rbac-proxy"
Jun 15 06:22:48.808803 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:48.808780    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2a66551-2406-4d19-8359-3853b091cab6" containerName="kube-rbac-proxy"
Jun 15 06:22:48.808803 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:48.808789    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2a66551-2406-4d19-8359-3853b091cab6" containerName="storage-initializer"
Jun 15 06:22:48.808803 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:48.808795    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2a66551-2406-4d19-8359-3853b091cab6" containerName="storage-initializer"
Jun 15 06:22:48.808995 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:48.808850    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2a66551-2406-4d19-8359-3853b091cab6" containerName="kube-rbac-proxy"
Jun 15 06:22:48.808995 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:48.808856    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2a66551-2406-4d19-8359-3853b091cab6" containerName="kserve-container"
Jun 15 06:22:48.812209 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:48.812193    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx"
Jun 15 06:22:48.814489 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:48.814466    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-452881-kube-rbac-proxy-sar-config\""
Jun 15 06:22:48.814628 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:48.814504    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-452881-dockercfg-66fcc\""
Jun 15 06:22:48.814628 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:48.814465    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-452881-predictor-serving-cert\""
Jun 15 06:22:48.814628 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:48.814575    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-452881\""
Jun 15 06:22:48.822727 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:48.822704    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx"]
Jun 15 06:22:48.883860 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:48.883830    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvtm4\" (UniqueName: \"kubernetes.io/projected/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-kube-api-access-tvtm4\") pod \"isvc-secondary-452881-predictor-6b59fd8f47-5vqwx\" (UID: \"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c\") " pod="kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx"
Jun 15 06:22:48.884032 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:48.883899    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-452881-predictor-6b59fd8f47-5vqwx\" (UID: \"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c\") " pod="kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx"
Jun 15 06:22:48.884032 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:48.883922    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-proxy-tls\") pod \"isvc-secondary-452881-predictor-6b59fd8f47-5vqwx\" (UID: \"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c\") " pod="kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx"
Jun 15 06:22:48.884032 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:48.883939    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-kserve-provision-location\") pod \"isvc-secondary-452881-predictor-6b59fd8f47-5vqwx\" (UID: \"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c\") " pod="kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx"
Jun 15 06:22:48.884032 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:48.883972    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-cabundle-cert\") pod \"isvc-secondary-452881-predictor-6b59fd8f47-5vqwx\" (UID: \"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c\") " pod="kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx"
Jun 15 06:22:48.985150 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:48.985116    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-452881-predictor-6b59fd8f47-5vqwx\" (UID: \"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c\") " pod="kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx"
Jun 15 06:22:48.985336 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:48.985157    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-proxy-tls\") pod \"isvc-secondary-452881-predictor-6b59fd8f47-5vqwx\" (UID: \"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c\") " pod="kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx"
Jun 15 06:22:48.985454 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:22:48.985332    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-452881-predictor-serving-cert: secret "isvc-secondary-452881-predictor-serving-cert" not found
Jun 15 06:22:48.985454 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:48.985408    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-kserve-provision-location\") pod \"isvc-secondary-452881-predictor-6b59fd8f47-5vqwx\" (UID: \"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c\") " pod="kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx"
Jun 15 06:22:48.985454 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:22:48.985429    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-proxy-tls podName:3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c nodeName:}" failed. No retries permitted until 2026-06-15 06:22:49.485407544 +0000 UTC m=+2084.633587519 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-proxy-tls") pod "isvc-secondary-452881-predictor-6b59fd8f47-5vqwx" (UID: "3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c") : secret "isvc-secondary-452881-predictor-serving-cert" not found
Jun 15 06:22:48.985626 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:48.985516    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-cabundle-cert\") pod \"isvc-secondary-452881-predictor-6b59fd8f47-5vqwx\" (UID: \"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c\") " pod="kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx"
Jun 15 06:22:48.985626 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:48.985562    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvtm4\" (UniqueName: \"kubernetes.io/projected/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-kube-api-access-tvtm4\") pod \"isvc-secondary-452881-predictor-6b59fd8f47-5vqwx\" (UID: \"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c\") " pod="kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx"
Jun 15 06:22:48.985730 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:48.985707    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-kserve-provision-location\") pod \"isvc-secondary-452881-predictor-6b59fd8f47-5vqwx\" (UID: \"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c\") " pod="kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx"
Jun 15 06:22:48.985869 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:48.985851    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-452881-predictor-6b59fd8f47-5vqwx\" (UID: \"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c\") " pod="kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx"
Jun 15 06:22:48.986064 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:48.986048    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-cabundle-cert\") pod \"isvc-secondary-452881-predictor-6b59fd8f47-5vqwx\" (UID: \"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c\") " pod="kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx"
Jun 15 06:22:48.995239 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:48.995212    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvtm4\" (UniqueName: \"kubernetes.io/projected/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-kube-api-access-tvtm4\") pod \"isvc-secondary-452881-predictor-6b59fd8f47-5vqwx\" (UID: \"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c\") " pod="kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx"
Jun 15 06:22:49.488781 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:49.488746    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-proxy-tls\") pod \"isvc-secondary-452881-predictor-6b59fd8f47-5vqwx\" (UID: \"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c\") " pod="kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx"
Jun 15 06:22:49.491220 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:49.491196    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-proxy-tls\") pod \"isvc-secondary-452881-predictor-6b59fd8f47-5vqwx\" (UID: \"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c\") " pod="kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx"
Jun 15 06:22:49.723262 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:49.723223    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx"
Jun 15 06:22:49.852873 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:49.852848    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx"]
Jun 15 06:22:49.854861 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:22:49.854822    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a6b6b2e_0f86_4d4c_8aa2_eb08e46d369c.slice/crio-0994d0fe70cf50fa5f87c85898b589d01656082cd95d25ef6322b852ce05f9b5 WatchSource:0}: Error finding container 0994d0fe70cf50fa5f87c85898b589d01656082cd95d25ef6322b852ce05f9b5: Status 404 returned error can't find the container with id 0994d0fe70cf50fa5f87c85898b589d01656082cd95d25ef6322b852ce05f9b5
Jun 15 06:22:50.364867 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:50.364833    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx" event={"ID":"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c","Type":"ContainerStarted","Data":"711e494a68c22421a37f9b8e9eea0a77d41d2c1e63e73b2d56ef0c24788ece24"}
Jun 15 06:22:50.365077 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:50.364889    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx" event={"ID":"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c","Type":"ContainerStarted","Data":"0994d0fe70cf50fa5f87c85898b589d01656082cd95d25ef6322b852ce05f9b5"}
Jun 15 06:22:57.398775 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:57.398747    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-452881-predictor-6b59fd8f47-5vqwx_3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c/storage-initializer/0.log"
Jun 15 06:22:57.399131 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:57.398784    2572 generic.go:356] "Generic (PLEG): container finished" podID="3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c" containerID="711e494a68c22421a37f9b8e9eea0a77d41d2c1e63e73b2d56ef0c24788ece24" exitCode=1
Jun 15 06:22:57.399131 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:57.398824    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx" event={"ID":"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c","Type":"ContainerDied","Data":"711e494a68c22421a37f9b8e9eea0a77d41d2c1e63e73b2d56ef0c24788ece24"}
Jun 15 06:22:58.404902 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:58.404874    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-452881-predictor-6b59fd8f47-5vqwx_3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c/storage-initializer/0.log"
Jun 15 06:22:58.405280 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:22:58.404950    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx" event={"ID":"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c","Type":"ContainerStarted","Data":"0ec087d43b40dd257fe2506a2c0c4e418c0a7922b0496dc4a3a6b5816b8751de"}
Jun 15 06:23:05.434878 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:05.434844    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-452881-predictor-6b59fd8f47-5vqwx_3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c/storage-initializer/1.log"
Jun 15 06:23:05.435284 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:05.435268    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-452881-predictor-6b59fd8f47-5vqwx_3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c/storage-initializer/0.log"
Jun 15 06:23:05.437621 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:05.437601    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-452881-predictor-6b59fd8f47-5vqwx_3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c/storage-initializer/1.log"
Jun 15 06:23:05.437974 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:05.437957    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-452881-predictor-6b59fd8f47-5vqwx_3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c/storage-initializer/0.log"
Jun 15 06:23:05.438076 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:05.437989    2572 generic.go:356] "Generic (PLEG): container finished" podID="3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c" containerID="0ec087d43b40dd257fe2506a2c0c4e418c0a7922b0496dc4a3a6b5816b8751de" exitCode=1
Jun 15 06:23:05.438076 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:05.438039    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx" event={"ID":"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c","Type":"ContainerDied","Data":"0ec087d43b40dd257fe2506a2c0c4e418c0a7922b0496dc4a3a6b5816b8751de"}
Jun 15 06:23:05.438203 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:05.438120    2572 scope.go:117] "RemoveContainer" containerID="711e494a68c22421a37f9b8e9eea0a77d41d2c1e63e73b2d56ef0c24788ece24"
Jun 15 06:23:05.438561 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:05.438538    2572 scope.go:117] "RemoveContainer" containerID="711e494a68c22421a37f9b8e9eea0a77d41d2c1e63e73b2d56ef0c24788ece24"
Jun 15 06:23:05.446189 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:05.446169    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-452881-predictor-6b59fd8f47-5vqwx_3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c/storage-initializer/1.log"
Jun 15 06:23:05.450486 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:23:05.450459    2572 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-452881-predictor-6b59fd8f47-5vqwx_kserve-ci-e2e-test(3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c)\"" pod="kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx" podUID="3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c"
Jun 15 06:23:05.450704 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:23:05.450673    2572 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-452881-predictor-6b59fd8f47-5vqwx_kserve-ci-e2e-test_3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c_0 in pod sandbox 0994d0fe70cf50fa5f87c85898b589d01656082cd95d25ef6322b852ce05f9b5 from index: no such id: '711e494a68c22421a37f9b8e9eea0a77d41d2c1e63e73b2d56ef0c24788ece24'" containerID="711e494a68c22421a37f9b8e9eea0a77d41d2c1e63e73b2d56ef0c24788ece24"
Jun 15 06:23:05.450808 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:05.450714    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"711e494a68c22421a37f9b8e9eea0a77d41d2c1e63e73b2d56ef0c24788ece24"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-452881-predictor-6b59fd8f47-5vqwx_kserve-ci-e2e-test_3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c_0 in pod sandbox 0994d0fe70cf50fa5f87c85898b589d01656082cd95d25ef6322b852ce05f9b5 from index: no such id: '711e494a68c22421a37f9b8e9eea0a77d41d2c1e63e73b2d56ef0c24788ece24'"
Jun 15 06:23:05.454194 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:05.454179    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 06:23:05.464460 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:05.464440    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 06:23:06.444542 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:06.444516    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-452881-predictor-6b59fd8f47-5vqwx_3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c/storage-initializer/1.log"
Jun 15 06:23:08.872890 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:08.872858    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx"]
Jun 15 06:23:08.922158 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:08.914713    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq"]
Jun 15 06:23:08.922158 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:08.915091    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq" podUID="f16264e1-fa3f-43e4-9909-ef67243c1c71" containerName="kserve-container" containerID="cri-o://1fb108b8930eb9035e469819defc8522e74a30bcf73d557b7aaf14a17b7c7711" gracePeriod=30
Jun 15 06:23:08.922158 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:08.915309    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq" podUID="f16264e1-fa3f-43e4-9909-ef67243c1c71" containerName="kube-rbac-proxy" containerID="cri-o://5ffc58de1236cf1c1bf3b998e5825bb92a9ac7c23cdacfaf7eecb0c4e614de5e" gracePeriod=30
Jun 15 06:23:08.994349 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:08.994317    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x"]
Jun 15 06:23:08.999177 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:08.999157    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x"
Jun 15 06:23:09.001308 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.001286    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-4315af-predictor-serving-cert\""
Jun 15 06:23:09.001435 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.001288    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-4315af\""
Jun 15 06:23:09.001435 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.001341    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-4315af-dockercfg-v4jjf\""
Jun 15 06:23:09.001737 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.001722    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-4315af-kube-rbac-proxy-sar-config\""
Jun 15 06:23:09.008477 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.008456    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x"]
Jun 15 06:23:09.045953 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.045934    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-452881-predictor-6b59fd8f47-5vqwx_3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c/storage-initializer/1.log"
Jun 15 06:23:09.047982 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.047964    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx"
Jun 15 06:23:09.165833 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.165804    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvtm4\" (UniqueName: \"kubernetes.io/projected/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-kube-api-access-tvtm4\") pod \"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c\" (UID: \"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c\") "
Jun 15 06:23:09.166031 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.165866    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-proxy-tls\") pod \"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c\" (UID: \"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c\") "
Jun 15 06:23:09.166031 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.165997    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-kserve-provision-location\") pod \"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c\" (UID: \"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c\") "
Jun 15 06:23:09.166169 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.166043    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-kube-rbac-proxy-sar-config\") pod \"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c\" (UID: \"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c\") "
Jun 15 06:23:09.166169 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.166129    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-cabundle-cert\") pod \"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c\" (UID: \"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c\") "
Jun 15 06:23:09.166336 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.166305    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c" (UID: "3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:23:09.166419 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.166329    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2csm9\" (UniqueName: \"kubernetes.io/projected/afec66e8-43ee-44cc-8e1a-28f00ef70570-kube-api-access-2csm9\") pod \"isvc-init-fail-4315af-predictor-549cf946d5-zxl8x\" (UID: \"afec66e8-43ee-44cc-8e1a-28f00ef70570\") " pod="kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x"
Jun 15 06:23:09.166419 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.166402    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afec66e8-43ee-44cc-8e1a-28f00ef70570-kserve-provision-location\") pod \"isvc-init-fail-4315af-predictor-549cf946d5-zxl8x\" (UID: \"afec66e8-43ee-44cc-8e1a-28f00ef70570\") " pod="kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x"
Jun 15 06:23:09.166536 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.166451    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c" (UID: "3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:23:09.166596 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.166564    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afec66e8-43ee-44cc-8e1a-28f00ef70570-cabundle-cert\") pod \"isvc-init-fail-4315af-predictor-549cf946d5-zxl8x\" (UID: \"afec66e8-43ee-44cc-8e1a-28f00ef70570\") " pod="kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x"
Jun 15 06:23:09.166657 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.166606    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afec66e8-43ee-44cc-8e1a-28f00ef70570-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-4315af-predictor-549cf946d5-zxl8x\" (UID: \"afec66e8-43ee-44cc-8e1a-28f00ef70570\") " pod="kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x"
Jun 15 06:23:09.166657 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.166622    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c" (UID: "3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:23:09.166768 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.166721    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afec66e8-43ee-44cc-8e1a-28f00ef70570-proxy-tls\") pod \"isvc-init-fail-4315af-predictor-549cf946d5-zxl8x\" (UID: \"afec66e8-43ee-44cc-8e1a-28f00ef70570\") " pod="kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x"
Jun 15 06:23:09.166820 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.166786    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:23:09.166820 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.166803    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:23:09.166931 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.166819    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:23:09.167786 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.167761    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-kube-api-access-tvtm4" (OuterVolumeSpecName: "kube-api-access-tvtm4") pod "3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c" (UID: "3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c"). InnerVolumeSpecName "kube-api-access-tvtm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:23:09.168169 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.168151    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c" (UID: "3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:23:09.267513 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.267474    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afec66e8-43ee-44cc-8e1a-28f00ef70570-cabundle-cert\") pod \"isvc-init-fail-4315af-predictor-549cf946d5-zxl8x\" (UID: \"afec66e8-43ee-44cc-8e1a-28f00ef70570\") " pod="kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x"
Jun 15 06:23:09.267513 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.267518    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afec66e8-43ee-44cc-8e1a-28f00ef70570-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-4315af-predictor-549cf946d5-zxl8x\" (UID: \"afec66e8-43ee-44cc-8e1a-28f00ef70570\") " pod="kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x"
Jun 15 06:23:09.267745 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.267564    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afec66e8-43ee-44cc-8e1a-28f00ef70570-proxy-tls\") pod \"isvc-init-fail-4315af-predictor-549cf946d5-zxl8x\" (UID: \"afec66e8-43ee-44cc-8e1a-28f00ef70570\") " pod="kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x"
Jun 15 06:23:09.267745 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.267583    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2csm9\" (UniqueName: \"kubernetes.io/projected/afec66e8-43ee-44cc-8e1a-28f00ef70570-kube-api-access-2csm9\") pod \"isvc-init-fail-4315af-predictor-549cf946d5-zxl8x\" (UID: \"afec66e8-43ee-44cc-8e1a-28f00ef70570\") " pod="kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x"
Jun 15 06:23:09.267745 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.267605    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afec66e8-43ee-44cc-8e1a-28f00ef70570-kserve-provision-location\") pod \"isvc-init-fail-4315af-predictor-549cf946d5-zxl8x\" (UID: \"afec66e8-43ee-44cc-8e1a-28f00ef70570\") " pod="kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x"
Jun 15 06:23:09.267745 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.267702    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvtm4\" (UniqueName: \"kubernetes.io/projected/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-kube-api-access-tvtm4\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:23:09.267745 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.267717    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:23:09.268084 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.268066    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afec66e8-43ee-44cc-8e1a-28f00ef70570-kserve-provision-location\") pod \"isvc-init-fail-4315af-predictor-549cf946d5-zxl8x\" (UID: \"afec66e8-43ee-44cc-8e1a-28f00ef70570\") " pod="kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x"
Jun 15 06:23:09.268277 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.268258    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afec66e8-43ee-44cc-8e1a-28f00ef70570-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-4315af-predictor-549cf946d5-zxl8x\" (UID: \"afec66e8-43ee-44cc-8e1a-28f00ef70570\") " pod="kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x"
Jun 15 06:23:09.268277 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.268271    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afec66e8-43ee-44cc-8e1a-28f00ef70570-cabundle-cert\") pod \"isvc-init-fail-4315af-predictor-549cf946d5-zxl8x\" (UID: \"afec66e8-43ee-44cc-8e1a-28f00ef70570\") " pod="kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x"
Jun 15 06:23:09.269986 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.269967    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afec66e8-43ee-44cc-8e1a-28f00ef70570-proxy-tls\") pod \"isvc-init-fail-4315af-predictor-549cf946d5-zxl8x\" (UID: \"afec66e8-43ee-44cc-8e1a-28f00ef70570\") " pod="kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x"
Jun 15 06:23:09.276168 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.276152    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2csm9\" (UniqueName: \"kubernetes.io/projected/afec66e8-43ee-44cc-8e1a-28f00ef70570-kube-api-access-2csm9\") pod \"isvc-init-fail-4315af-predictor-549cf946d5-zxl8x\" (UID: \"afec66e8-43ee-44cc-8e1a-28f00ef70570\") " pod="kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x"
Jun 15 06:23:09.316088 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.316063    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x"
Jun 15 06:23:09.451631 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.451579    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x"]
Jun 15 06:23:09.453759 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:23:09.453728    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafec66e8_43ee_44cc_8e1a_28f00ef70570.slice/crio-4f63819802c2570bb000f27ca7f733d877546de1702acdf0733b5b3638894458 WatchSource:0}: Error finding container 4f63819802c2570bb000f27ca7f733d877546de1702acdf0733b5b3638894458: Status 404 returned error can't find the container with id 4f63819802c2570bb000f27ca7f733d877546de1702acdf0733b5b3638894458
Jun 15 06:23:09.459919 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.459896    2572 generic.go:356] "Generic (PLEG): container finished" podID="f16264e1-fa3f-43e4-9909-ef67243c1c71" containerID="5ffc58de1236cf1c1bf3b998e5825bb92a9ac7c23cdacfaf7eecb0c4e614de5e" exitCode=2
Jun 15 06:23:09.460000 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.459966    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq" event={"ID":"f16264e1-fa3f-43e4-9909-ef67243c1c71","Type":"ContainerDied","Data":"5ffc58de1236cf1c1bf3b998e5825bb92a9ac7c23cdacfaf7eecb0c4e614de5e"}
Jun 15 06:23:09.461145 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.461130    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-452881-predictor-6b59fd8f47-5vqwx_3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c/storage-initializer/1.log"
Jun 15 06:23:09.461196 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.461187    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx" event={"ID":"3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c","Type":"ContainerDied","Data":"0994d0fe70cf50fa5f87c85898b589d01656082cd95d25ef6322b852ce05f9b5"}
Jun 15 06:23:09.461257 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.461231    2572 scope.go:117] "RemoveContainer" containerID="0ec087d43b40dd257fe2506a2c0c4e418c0a7922b0496dc4a3a6b5816b8751de"
Jun 15 06:23:09.461314 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.461228    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx"
Jun 15 06:23:09.514662 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.514638    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx"]
Jun 15 06:23:09.528461 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:09.528434    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-452881-predictor-6b59fd8f47-5vqwx"]
Jun 15 06:23:10.319624 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:10.319583    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq" podUID="f16264e1-fa3f-43e4-9909-ef67243c1c71" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jun 15 06:23:10.468544 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:10.468510    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x" event={"ID":"afec66e8-43ee-44cc-8e1a-28f00ef70570","Type":"ContainerStarted","Data":"1c5d3e53a08f75a393e982531547ef34c5e6aab947e3d40630d428146dea735c"}
Jun 15 06:23:10.468725 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:10.468569    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x" event={"ID":"afec66e8-43ee-44cc-8e1a-28f00ef70570","Type":"ContainerStarted","Data":"4f63819802c2570bb000f27ca7f733d877546de1702acdf0733b5b3638894458"}
Jun 15 06:23:10.974433 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:10.974388    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq" podUID="f16264e1-fa3f-43e4-9909-ef67243c1c71" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Jun 15 06:23:11.324208 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:11.324129    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c" path="/var/lib/kubelet/pods/3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c/volumes"
Jun 15 06:23:11.474301 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:11.474274    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4315af-predictor-549cf946d5-zxl8x_afec66e8-43ee-44cc-8e1a-28f00ef70570/storage-initializer/0.log"
Jun 15 06:23:11.474497 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:11.474317    2572 generic.go:356] "Generic (PLEG): container finished" podID="afec66e8-43ee-44cc-8e1a-28f00ef70570" containerID="1c5d3e53a08f75a393e982531547ef34c5e6aab947e3d40630d428146dea735c" exitCode=1
Jun 15 06:23:11.474497 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:11.474404    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x" event={"ID":"afec66e8-43ee-44cc-8e1a-28f00ef70570","Type":"ContainerDied","Data":"1c5d3e53a08f75a393e982531547ef34c5e6aab947e3d40630d428146dea735c"}
Jun 15 06:23:12.481030 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:12.481000    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4315af-predictor-549cf946d5-zxl8x_afec66e8-43ee-44cc-8e1a-28f00ef70570/storage-initializer/0.log"
Jun 15 06:23:12.481473 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:12.481131    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x" event={"ID":"afec66e8-43ee-44cc-8e1a-28f00ef70570","Type":"ContainerStarted","Data":"0b9ffd9151dde630883f54b19abb00c1487dd0ee6a901e7caaf91189c5e13d19"}
Jun 15 06:23:13.665268 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:13.665242    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq"
Jun 15 06:23:13.702372 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:13.702284    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f16264e1-fa3f-43e4-9909-ef67243c1c71-kserve-provision-location\") pod \"f16264e1-fa3f-43e4-9909-ef67243c1c71\" (UID: \"f16264e1-fa3f-43e4-9909-ef67243c1c71\") "
Jun 15 06:23:13.702372 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:13.702323    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f16264e1-fa3f-43e4-9909-ef67243c1c71-cabundle-cert\") pod \"f16264e1-fa3f-43e4-9909-ef67243c1c71\" (UID: \"f16264e1-fa3f-43e4-9909-ef67243c1c71\") "
Jun 15 06:23:13.702597 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:13.702392    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f16264e1-fa3f-43e4-9909-ef67243c1c71-kube-rbac-proxy-sar-config\") pod \"f16264e1-fa3f-43e4-9909-ef67243c1c71\" (UID: \"f16264e1-fa3f-43e4-9909-ef67243c1c71\") "
Jun 15 06:23:13.702597 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:13.702435    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rst96\" (UniqueName: \"kubernetes.io/projected/f16264e1-fa3f-43e4-9909-ef67243c1c71-kube-api-access-rst96\") pod \"f16264e1-fa3f-43e4-9909-ef67243c1c71\" (UID: \"f16264e1-fa3f-43e4-9909-ef67243c1c71\") "
Jun 15 06:23:13.702597 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:13.702516    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f16264e1-fa3f-43e4-9909-ef67243c1c71-proxy-tls\") pod \"f16264e1-fa3f-43e4-9909-ef67243c1c71\" (UID: \"f16264e1-fa3f-43e4-9909-ef67243c1c71\") "
Jun 15 06:23:13.702744 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:13.702684    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f16264e1-fa3f-43e4-9909-ef67243c1c71-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f16264e1-fa3f-43e4-9909-ef67243c1c71" (UID: "f16264e1-fa3f-43e4-9909-ef67243c1c71"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:23:13.702817 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:13.702801    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f16264e1-fa3f-43e4-9909-ef67243c1c71-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:23:13.702912 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:13.702799    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f16264e1-fa3f-43e4-9909-ef67243c1c71-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f16264e1-fa3f-43e4-9909-ef67243c1c71" (UID: "f16264e1-fa3f-43e4-9909-ef67243c1c71"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:23:13.702912 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:13.702821    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f16264e1-fa3f-43e4-9909-ef67243c1c71-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f16264e1-fa3f-43e4-9909-ef67243c1c71" (UID: "f16264e1-fa3f-43e4-9909-ef67243c1c71"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:23:13.704601 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:13.704577    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f16264e1-fa3f-43e4-9909-ef67243c1c71-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f16264e1-fa3f-43e4-9909-ef67243c1c71" (UID: "f16264e1-fa3f-43e4-9909-ef67243c1c71"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:23:13.704702 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:13.704624    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f16264e1-fa3f-43e4-9909-ef67243c1c71-kube-api-access-rst96" (OuterVolumeSpecName: "kube-api-access-rst96") pod "f16264e1-fa3f-43e4-9909-ef67243c1c71" (UID: "f16264e1-fa3f-43e4-9909-ef67243c1c71"). InnerVolumeSpecName "kube-api-access-rst96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:23:13.804103 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:13.804053    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f16264e1-fa3f-43e4-9909-ef67243c1c71-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:23:13.804103 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:13.804099    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f16264e1-fa3f-43e4-9909-ef67243c1c71-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:23:13.804103 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:13.804110    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f16264e1-fa3f-43e4-9909-ef67243c1c71-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:23:13.804103 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:13.804120    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rst96\" (UniqueName: \"kubernetes.io/projected/f16264e1-fa3f-43e4-9909-ef67243c1c71-kube-api-access-rst96\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:23:14.004590 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.004484    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x"]
Jun 15 06:23:14.434023 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.433991    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v"]
Jun 15 06:23:14.434390 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.434353    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f16264e1-fa3f-43e4-9909-ef67243c1c71" containerName="kube-rbac-proxy"
Jun 15 06:23:14.434390 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.434383    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f16264e1-fa3f-43e4-9909-ef67243c1c71" containerName="kube-rbac-proxy"
Jun 15 06:23:14.434390 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.434393    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c" containerName="storage-initializer"
Jun 15 06:23:14.434640 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.434401    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c" containerName="storage-initializer"
Jun 15 06:23:14.434640 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.434411    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c" containerName="storage-initializer"
Jun 15 06:23:14.434640 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.434419    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c" containerName="storage-initializer"
Jun 15 06:23:14.434640 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.434436    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f16264e1-fa3f-43e4-9909-ef67243c1c71" containerName="storage-initializer"
Jun 15 06:23:14.434640 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.434444    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f16264e1-fa3f-43e4-9909-ef67243c1c71" containerName="storage-initializer"
Jun 15 06:23:14.434640 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.434455    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f16264e1-fa3f-43e4-9909-ef67243c1c71" containerName="kserve-container"
Jun 15 06:23:14.434640 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.434463    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f16264e1-fa3f-43e4-9909-ef67243c1c71" containerName="kserve-container"
Jun 15 06:23:14.434640 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.434552    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c" containerName="storage-initializer"
Jun 15 06:23:14.434640 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.434566    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f16264e1-fa3f-43e4-9909-ef67243c1c71" containerName="kserve-container"
Jun 15 06:23:14.434640 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.434576    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f16264e1-fa3f-43e4-9909-ef67243c1c71" containerName="kube-rbac-proxy"
Jun 15 06:23:14.435119 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.434753    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a6b6b2e-0f86-4d4c-8aa2-eb08e46d369c" containerName="storage-initializer"
Jun 15 06:23:14.438079 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.438061    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v"
Jun 15 06:23:14.440629 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.440607    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Jun 15 06:23:14.440748 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.440726    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Jun 15 06:23:14.447777 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.447751    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v"]
Jun 15 06:23:14.492118 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.492091    2572 generic.go:356] "Generic (PLEG): container finished" podID="f16264e1-fa3f-43e4-9909-ef67243c1c71" containerID="1fb108b8930eb9035e469819defc8522e74a30bcf73d557b7aaf14a17b7c7711" exitCode=0
Jun 15 06:23:14.492244 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.492151    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq"
Jun 15 06:23:14.492244 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.492170    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq" event={"ID":"f16264e1-fa3f-43e4-9909-ef67243c1c71","Type":"ContainerDied","Data":"1fb108b8930eb9035e469819defc8522e74a30bcf73d557b7aaf14a17b7c7711"}
Jun 15 06:23:14.492244 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.492223    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq" event={"ID":"f16264e1-fa3f-43e4-9909-ef67243c1c71","Type":"ContainerDied","Data":"54cfbe79d2ba299396475d86f102588b2802dc503037cd9e46360491eb7eb842"}
Jun 15 06:23:14.492351 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.492252    2572 scope.go:117] "RemoveContainer" containerID="5ffc58de1236cf1c1bf3b998e5825bb92a9ac7c23cdacfaf7eecb0c4e614de5e"
Jun 15 06:23:14.492506 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.492477    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x" podUID="afec66e8-43ee-44cc-8e1a-28f00ef70570" containerName="storage-initializer" containerID="cri-o://0b9ffd9151dde630883f54b19abb00c1487dd0ee6a901e7caaf91189c5e13d19" gracePeriod=30
Jun 15 06:23:14.501490 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.501466    2572 scope.go:117] "RemoveContainer" containerID="1fb108b8930eb9035e469819defc8522e74a30bcf73d557b7aaf14a17b7c7711"
Jun 15 06:23:14.509654 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.509637    2572 scope.go:117] "RemoveContainer" containerID="5c187434aa2af003850ad3fb80f4944b4477cc75987c1cd025a1dfe2a8d49a63"
Jun 15 06:23:14.509929 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.509911    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44a313fb-c1e6-4df8-b000-ea83697f6431-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-5c69v\" (UID: \"44a313fb-c1e6-4df8-b000-ea83697f6431\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v"
Jun 15 06:23:14.510018 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.509936    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44a313fb-c1e6-4df8-b000-ea83697f6431-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-5c69v\" (UID: \"44a313fb-c1e6-4df8-b000-ea83697f6431\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v"
Jun 15 06:23:14.510018 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.509959    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44a313fb-c1e6-4df8-b000-ea83697f6431-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-5c69v\" (UID: \"44a313fb-c1e6-4df8-b000-ea83697f6431\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v"
Jun 15 06:23:14.510018 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.509998    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44a313fb-c1e6-4df8-b000-ea83697f6431-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-5c69v\" (UID: \"44a313fb-c1e6-4df8-b000-ea83697f6431\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v"
Jun 15 06:23:14.510182 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.510051    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgj7b\" (UniqueName: \"kubernetes.io/projected/44a313fb-c1e6-4df8-b000-ea83697f6431-kube-api-access-lgj7b\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-5c69v\" (UID: \"44a313fb-c1e6-4df8-b000-ea83697f6431\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v"
Jun 15 06:23:14.518935 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.518886    2572 scope.go:117] "RemoveContainer" containerID="5ffc58de1236cf1c1bf3b998e5825bb92a9ac7c23cdacfaf7eecb0c4e614de5e"
Jun 15 06:23:14.519404 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:23:14.519341    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ffc58de1236cf1c1bf3b998e5825bb92a9ac7c23cdacfaf7eecb0c4e614de5e\": container with ID starting with 5ffc58de1236cf1c1bf3b998e5825bb92a9ac7c23cdacfaf7eecb0c4e614de5e not found: ID does not exist" containerID="5ffc58de1236cf1c1bf3b998e5825bb92a9ac7c23cdacfaf7eecb0c4e614de5e"
Jun 15 06:23:14.519502 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.519426    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ffc58de1236cf1c1bf3b998e5825bb92a9ac7c23cdacfaf7eecb0c4e614de5e"} err="failed to get container status \"5ffc58de1236cf1c1bf3b998e5825bb92a9ac7c23cdacfaf7eecb0c4e614de5e\": rpc error: code = NotFound desc = could not find container \"5ffc58de1236cf1c1bf3b998e5825bb92a9ac7c23cdacfaf7eecb0c4e614de5e\": container with ID starting with 5ffc58de1236cf1c1bf3b998e5825bb92a9ac7c23cdacfaf7eecb0c4e614de5e not found: ID does not exist"
Jun 15 06:23:14.519502 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.519445    2572 scope.go:117] "RemoveContainer" containerID="1fb108b8930eb9035e469819defc8522e74a30bcf73d557b7aaf14a17b7c7711"
Jun 15 06:23:14.519710 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:23:14.519692    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fb108b8930eb9035e469819defc8522e74a30bcf73d557b7aaf14a17b7c7711\": container with ID starting with 1fb108b8930eb9035e469819defc8522e74a30bcf73d557b7aaf14a17b7c7711 not found: ID does not exist" containerID="1fb108b8930eb9035e469819defc8522e74a30bcf73d557b7aaf14a17b7c7711"
Jun 15 06:23:14.519770 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.519720    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fb108b8930eb9035e469819defc8522e74a30bcf73d557b7aaf14a17b7c7711"} err="failed to get container status \"1fb108b8930eb9035e469819defc8522e74a30bcf73d557b7aaf14a17b7c7711\": rpc error: code = NotFound desc = could not find container \"1fb108b8930eb9035e469819defc8522e74a30bcf73d557b7aaf14a17b7c7711\": container with ID starting with 1fb108b8930eb9035e469819defc8522e74a30bcf73d557b7aaf14a17b7c7711 not found: ID does not exist"
Jun 15 06:23:14.519770 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.519737    2572 scope.go:117] "RemoveContainer" containerID="5c187434aa2af003850ad3fb80f4944b4477cc75987c1cd025a1dfe2a8d49a63"
Jun 15 06:23:14.519994 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:23:14.519977    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c187434aa2af003850ad3fb80f4944b4477cc75987c1cd025a1dfe2a8d49a63\": container with ID starting with 5c187434aa2af003850ad3fb80f4944b4477cc75987c1cd025a1dfe2a8d49a63 not found: ID does not exist" containerID="5c187434aa2af003850ad3fb80f4944b4477cc75987c1cd025a1dfe2a8d49a63"
Jun 15 06:23:14.520051 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.519999    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c187434aa2af003850ad3fb80f4944b4477cc75987c1cd025a1dfe2a8d49a63"} err="failed to get container status \"5c187434aa2af003850ad3fb80f4944b4477cc75987c1cd025a1dfe2a8d49a63\": rpc error: code = NotFound desc = could not find container \"5c187434aa2af003850ad3fb80f4944b4477cc75987c1cd025a1dfe2a8d49a63\": container with ID starting with 5c187434aa2af003850ad3fb80f4944b4477cc75987c1cd025a1dfe2a8d49a63 not found: ID does not exist"
Jun 15 06:23:14.546816 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.546796    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq"]
Jun 15 06:23:14.553054 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.553030    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-452881-predictor-6bd5b98f4d-sskgq"]
Jun 15 06:23:14.610657 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.610631    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44a313fb-c1e6-4df8-b000-ea83697f6431-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-5c69v\" (UID: \"44a313fb-c1e6-4df8-b000-ea83697f6431\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v"
Jun 15 06:23:14.610799 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.610664    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44a313fb-c1e6-4df8-b000-ea83697f6431-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-5c69v\" (UID: \"44a313fb-c1e6-4df8-b000-ea83697f6431\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v"
Jun 15 06:23:14.610799 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.610698    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44a313fb-c1e6-4df8-b000-ea83697f6431-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-5c69v\" (UID: \"44a313fb-c1e6-4df8-b000-ea83697f6431\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v"
Jun 15 06:23:14.610799 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.610726    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44a313fb-c1e6-4df8-b000-ea83697f6431-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-5c69v\" (UID: \"44a313fb-c1e6-4df8-b000-ea83697f6431\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v"
Jun 15 06:23:14.610799 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.610769    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgj7b\" (UniqueName: \"kubernetes.io/projected/44a313fb-c1e6-4df8-b000-ea83697f6431-kube-api-access-lgj7b\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-5c69v\" (UID: \"44a313fb-c1e6-4df8-b000-ea83697f6431\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v"
Jun 15 06:23:14.611167 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.611144    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44a313fb-c1e6-4df8-b000-ea83697f6431-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-5c69v\" (UID: \"44a313fb-c1e6-4df8-b000-ea83697f6431\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v"
Jun 15 06:23:14.611389 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.611353    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44a313fb-c1e6-4df8-b000-ea83697f6431-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-5c69v\" (UID: \"44a313fb-c1e6-4df8-b000-ea83697f6431\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v"
Jun 15 06:23:14.611448 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.611399    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44a313fb-c1e6-4df8-b000-ea83697f6431-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-5c69v\" (UID: \"44a313fb-c1e6-4df8-b000-ea83697f6431\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v"
Jun 15 06:23:14.613125 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.613106    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44a313fb-c1e6-4df8-b000-ea83697f6431-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-5c69v\" (UID: \"44a313fb-c1e6-4df8-b000-ea83697f6431\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v"
Jun 15 06:23:14.621759 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.621730    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgj7b\" (UniqueName: \"kubernetes.io/projected/44a313fb-c1e6-4df8-b000-ea83697f6431-kube-api-access-lgj7b\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-5c69v\" (UID: \"44a313fb-c1e6-4df8-b000-ea83697f6431\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v"
Jun 15 06:23:14.749305 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.749229    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v"
Jun 15 06:23:14.883053 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:14.883027    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v"]
Jun 15 06:23:14.885374 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:23:14.885319    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a313fb_c1e6_4df8_b000_ea83697f6431.slice/crio-180a57667fe7757127079804e371c2de06140c132a3241c8a17d32fac23adeab WatchSource:0}: Error finding container 180a57667fe7757127079804e371c2de06140c132a3241c8a17d32fac23adeab: Status 404 returned error can't find the container with id 180a57667fe7757127079804e371c2de06140c132a3241c8a17d32fac23adeab
Jun 15 06:23:15.325634 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:15.325602    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f16264e1-fa3f-43e4-9909-ef67243c1c71" path="/var/lib/kubelet/pods/f16264e1-fa3f-43e4-9909-ef67243c1c71/volumes"
Jun 15 06:23:15.498499 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:15.498458    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v" event={"ID":"44a313fb-c1e6-4df8-b000-ea83697f6431","Type":"ContainerStarted","Data":"f24179180e56a7c1cbc71873fa648a38787dd2450d07aed108fe88c858daedd8"}
Jun 15 06:23:15.498706 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:15.498529    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v" event={"ID":"44a313fb-c1e6-4df8-b000-ea83697f6431","Type":"ContainerStarted","Data":"180a57667fe7757127079804e371c2de06140c132a3241c8a17d32fac23adeab"}
Jun 15 06:23:16.137934 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.137913    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4315af-predictor-549cf946d5-zxl8x_afec66e8-43ee-44cc-8e1a-28f00ef70570/storage-initializer/1.log"
Jun 15 06:23:16.138324 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.138308    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4315af-predictor-549cf946d5-zxl8x_afec66e8-43ee-44cc-8e1a-28f00ef70570/storage-initializer/0.log"
Jun 15 06:23:16.140626 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.140613    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x"
Jun 15 06:23:16.226046 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.225967    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2csm9\" (UniqueName: \"kubernetes.io/projected/afec66e8-43ee-44cc-8e1a-28f00ef70570-kube-api-access-2csm9\") pod \"afec66e8-43ee-44cc-8e1a-28f00ef70570\" (UID: \"afec66e8-43ee-44cc-8e1a-28f00ef70570\") "
Jun 15 06:23:16.226046 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.226000    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afec66e8-43ee-44cc-8e1a-28f00ef70570-cabundle-cert\") pod \"afec66e8-43ee-44cc-8e1a-28f00ef70570\" (UID: \"afec66e8-43ee-44cc-8e1a-28f00ef70570\") "
Jun 15 06:23:16.226046 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.226035    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afec66e8-43ee-44cc-8e1a-28f00ef70570-kserve-provision-location\") pod \"afec66e8-43ee-44cc-8e1a-28f00ef70570\" (UID: \"afec66e8-43ee-44cc-8e1a-28f00ef70570\") "
Jun 15 06:23:16.226284 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.226116    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afec66e8-43ee-44cc-8e1a-28f00ef70570-proxy-tls\") pod \"afec66e8-43ee-44cc-8e1a-28f00ef70570\" (UID: \"afec66e8-43ee-44cc-8e1a-28f00ef70570\") "
Jun 15 06:23:16.226284 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.226200    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afec66e8-43ee-44cc-8e1a-28f00ef70570-kube-rbac-proxy-sar-config\") pod \"afec66e8-43ee-44cc-8e1a-28f00ef70570\" (UID: \"afec66e8-43ee-44cc-8e1a-28f00ef70570\") "
Jun 15 06:23:16.226386 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.226303    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afec66e8-43ee-44cc-8e1a-28f00ef70570-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "afec66e8-43ee-44cc-8e1a-28f00ef70570" (UID: "afec66e8-43ee-44cc-8e1a-28f00ef70570"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:23:16.226531 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.226507    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afec66e8-43ee-44cc-8e1a-28f00ef70570-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "afec66e8-43ee-44cc-8e1a-28f00ef70570" (UID: "afec66e8-43ee-44cc-8e1a-28f00ef70570"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:23:16.226619 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.226546    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/afec66e8-43ee-44cc-8e1a-28f00ef70570-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:23:16.226619 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.226569    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afec66e8-43ee-44cc-8e1a-28f00ef70570-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "afec66e8-43ee-44cc-8e1a-28f00ef70570" (UID: "afec66e8-43ee-44cc-8e1a-28f00ef70570"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:23:16.228205 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.228184    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afec66e8-43ee-44cc-8e1a-28f00ef70570-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "afec66e8-43ee-44cc-8e1a-28f00ef70570" (UID: "afec66e8-43ee-44cc-8e1a-28f00ef70570"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:23:16.228324 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.228304    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afec66e8-43ee-44cc-8e1a-28f00ef70570-kube-api-access-2csm9" (OuterVolumeSpecName: "kube-api-access-2csm9") pod "afec66e8-43ee-44cc-8e1a-28f00ef70570" (UID: "afec66e8-43ee-44cc-8e1a-28f00ef70570"). InnerVolumeSpecName "kube-api-access-2csm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:23:16.327087 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.327045    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/afec66e8-43ee-44cc-8e1a-28f00ef70570-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:23:16.327087 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.327084    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2csm9\" (UniqueName: \"kubernetes.io/projected/afec66e8-43ee-44cc-8e1a-28f00ef70570-kube-api-access-2csm9\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:23:16.327087 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.327095    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/afec66e8-43ee-44cc-8e1a-28f00ef70570-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:23:16.327087 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.327105    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/afec66e8-43ee-44cc-8e1a-28f00ef70570-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:23:16.505259 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.505188    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4315af-predictor-549cf946d5-zxl8x_afec66e8-43ee-44cc-8e1a-28f00ef70570/storage-initializer/1.log"
Jun 15 06:23:16.505609 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.505595    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-4315af-predictor-549cf946d5-zxl8x_afec66e8-43ee-44cc-8e1a-28f00ef70570/storage-initializer/0.log"
Jun 15 06:23:16.505672 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.505631    2572 generic.go:356] "Generic (PLEG): container finished" podID="afec66e8-43ee-44cc-8e1a-28f00ef70570" containerID="0b9ffd9151dde630883f54b19abb00c1487dd0ee6a901e7caaf91189c5e13d19" exitCode=1
Jun 15 06:23:16.505707 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.505697    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x"
Jun 15 06:23:16.505744 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.505718    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x" event={"ID":"afec66e8-43ee-44cc-8e1a-28f00ef70570","Type":"ContainerDied","Data":"0b9ffd9151dde630883f54b19abb00c1487dd0ee6a901e7caaf91189c5e13d19"}
Jun 15 06:23:16.505792 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.505778    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x" event={"ID":"afec66e8-43ee-44cc-8e1a-28f00ef70570","Type":"ContainerDied","Data":"4f63819802c2570bb000f27ca7f733d877546de1702acdf0733b5b3638894458"}
Jun 15 06:23:16.505825 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.505798    2572 scope.go:117] "RemoveContainer" containerID="0b9ffd9151dde630883f54b19abb00c1487dd0ee6a901e7caaf91189c5e13d19"
Jun 15 06:23:16.514578 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.514560    2572 scope.go:117] "RemoveContainer" containerID="1c5d3e53a08f75a393e982531547ef34c5e6aab947e3d40630d428146dea735c"
Jun 15 06:23:16.522247 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.522229    2572 scope.go:117] "RemoveContainer" containerID="0b9ffd9151dde630883f54b19abb00c1487dd0ee6a901e7caaf91189c5e13d19"
Jun 15 06:23:16.522543 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:23:16.522522    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b9ffd9151dde630883f54b19abb00c1487dd0ee6a901e7caaf91189c5e13d19\": container with ID starting with 0b9ffd9151dde630883f54b19abb00c1487dd0ee6a901e7caaf91189c5e13d19 not found: ID does not exist" containerID="0b9ffd9151dde630883f54b19abb00c1487dd0ee6a901e7caaf91189c5e13d19"
Jun 15 06:23:16.522606 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.522551    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b9ffd9151dde630883f54b19abb00c1487dd0ee6a901e7caaf91189c5e13d19"} err="failed to get container status \"0b9ffd9151dde630883f54b19abb00c1487dd0ee6a901e7caaf91189c5e13d19\": rpc error: code = NotFound desc = could not find container \"0b9ffd9151dde630883f54b19abb00c1487dd0ee6a901e7caaf91189c5e13d19\": container with ID starting with 0b9ffd9151dde630883f54b19abb00c1487dd0ee6a901e7caaf91189c5e13d19 not found: ID does not exist"
Jun 15 06:23:16.522606 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.522569    2572 scope.go:117] "RemoveContainer" containerID="1c5d3e53a08f75a393e982531547ef34c5e6aab947e3d40630d428146dea735c"
Jun 15 06:23:16.522778 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:23:16.522764    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c5d3e53a08f75a393e982531547ef34c5e6aab947e3d40630d428146dea735c\": container with ID starting with 1c5d3e53a08f75a393e982531547ef34c5e6aab947e3d40630d428146dea735c not found: ID does not exist" containerID="1c5d3e53a08f75a393e982531547ef34c5e6aab947e3d40630d428146dea735c"
Jun 15 06:23:16.522819 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.522780    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c5d3e53a08f75a393e982531547ef34c5e6aab947e3d40630d428146dea735c"} err="failed to get container status \"1c5d3e53a08f75a393e982531547ef34c5e6aab947e3d40630d428146dea735c\": rpc error: code = NotFound desc = could not find container \"1c5d3e53a08f75a393e982531547ef34c5e6aab947e3d40630d428146dea735c\": container with ID starting with 1c5d3e53a08f75a393e982531547ef34c5e6aab947e3d40630d428146dea735c not found: ID does not exist"
Jun 15 06:23:16.546965 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.546835    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x"]
Jun 15 06:23:16.552939 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:16.552910    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-4315af-predictor-549cf946d5-zxl8x"]
Jun 15 06:23:17.324826 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:17.324789    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afec66e8-43ee-44cc-8e1a-28f00ef70570" path="/var/lib/kubelet/pods/afec66e8-43ee-44cc-8e1a-28f00ef70570/volumes"
Jun 15 06:23:19.521163 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:19.521132    2572 generic.go:356] "Generic (PLEG): container finished" podID="44a313fb-c1e6-4df8-b000-ea83697f6431" containerID="f24179180e56a7c1cbc71873fa648a38787dd2450d07aed108fe88c858daedd8" exitCode=0
Jun 15 06:23:19.521580 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:19.521216    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v" event={"ID":"44a313fb-c1e6-4df8-b000-ea83697f6431","Type":"ContainerDied","Data":"f24179180e56a7c1cbc71873fa648a38787dd2450d07aed108fe88c858daedd8"}
Jun 15 06:23:19.522467 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:19.522450    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 15 06:23:44.680131 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:44.680091    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v" event={"ID":"44a313fb-c1e6-4df8-b000-ea83697f6431","Type":"ContainerStarted","Data":"85ecbaf99260424ce7876e5a3c9896cf7c85f4fc07c43a6253f34e407d1f30de"}
Jun 15 06:23:44.680544 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:44.680164    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v" event={"ID":"44a313fb-c1e6-4df8-b000-ea83697f6431","Type":"ContainerStarted","Data":"c46e46a11ce12d6fa61680a0c8f9b1ba2954767a6f6b9a27796aaedd5521d1c7"}
Jun 15 06:23:44.680544 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:44.680428    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v"
Jun 15 06:23:44.680544 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:44.680475    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v"
Jun 15 06:23:44.681707 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:44.681682    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v" podUID="44a313fb-c1e6-4df8-b000-ea83697f6431" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jun 15 06:23:44.700346 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:44.700303    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v" podStartSLOduration=2.847470421 podStartE2EDuration="30.700289117s" totalImagesPullingTime="24.086285087s" totalInitContainerRuntime="3.766533609s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:23:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:23:44.699172363 +0000 UTC m=+2139.847352359" watchObservedRunningTime="2026-06-15 06:23:44.700289117 +0000 UTC m=+2139.848469117"
Jun 15 06:23:45.684667 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:45.684625    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v" podUID="44a313fb-c1e6-4df8-b000-ea83697f6431" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jun 15 06:23:50.689381 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:50.689341    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v"
Jun 15 06:23:50.689798 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:23:50.689769    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v" podUID="44a313fb-c1e6-4df8-b000-ea83697f6431" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jun 15 06:24:00.690351 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:24:00.690309    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v" podUID="44a313fb-c1e6-4df8-b000-ea83697f6431" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jun 15 06:24:10.690779 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:24:10.690743    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v" podUID="44a313fb-c1e6-4df8-b000-ea83697f6431" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jun 15 06:24:20.690146 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:24:20.690102    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v" podUID="44a313fb-c1e6-4df8-b000-ea83697f6431" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jun 15 06:24:30.690448 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:24:30.690409    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v" podUID="44a313fb-c1e6-4df8-b000-ea83697f6431" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jun 15 06:24:40.690234 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:24:40.690190    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v" podUID="44a313fb-c1e6-4df8-b000-ea83697f6431" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jun 15 06:24:50.689858 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:24:50.689813    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v" podUID="44a313fb-c1e6-4df8-b000-ea83697f6431" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jun 15 06:25:00.690565 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:00.690534    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v"
Jun 15 06:25:04.417909 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.417868    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v"]
Jun 15 06:25:04.418385 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.418185    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v" podUID="44a313fb-c1e6-4df8-b000-ea83697f6431" containerName="kserve-container" containerID="cri-o://c46e46a11ce12d6fa61680a0c8f9b1ba2954767a6f6b9a27796aaedd5521d1c7" gracePeriod=30
Jun 15 06:25:04.418385 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.418209    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v" podUID="44a313fb-c1e6-4df8-b000-ea83697f6431" containerName="kube-rbac-proxy" containerID="cri-o://85ecbaf99260424ce7876e5a3c9896cf7c85f4fc07c43a6253f34e407d1f30de" gracePeriod=30
Jun 15 06:25:04.512410 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.512353    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj"]
Jun 15 06:25:04.512840 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.512826    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afec66e8-43ee-44cc-8e1a-28f00ef70570" containerName="storage-initializer"
Jun 15 06:25:04.512890 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.512843    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="afec66e8-43ee-44cc-8e1a-28f00ef70570" containerName="storage-initializer"
Jun 15 06:25:04.512926 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.512918    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="afec66e8-43ee-44cc-8e1a-28f00ef70570" containerName="storage-initializer"
Jun 15 06:25:04.512961 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.512927    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="afec66e8-43ee-44cc-8e1a-28f00ef70570" containerName="storage-initializer"
Jun 15 06:25:04.512996 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.512984    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afec66e8-43ee-44cc-8e1a-28f00ef70570" containerName="storage-initializer"
Jun 15 06:25:04.512996 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.512990    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="afec66e8-43ee-44cc-8e1a-28f00ef70570" containerName="storage-initializer"
Jun 15 06:25:04.515504 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.515484    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj"
Jun 15 06:25:04.517627 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.517604    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Jun 15 06:25:04.517725 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.517653    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Jun 15 06:25:04.528567 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.528540    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj"]
Jun 15 06:25:04.632810 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.632772    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbkxz\" (UniqueName: \"kubernetes.io/projected/7472acfb-8091-4231-b490-6e885e3fc3bb-kube-api-access-tbkxz\") pod \"isvc-predictive-xgboost-predictor-695f978978-dkncj\" (UID: \"7472acfb-8091-4231-b490-6e885e3fc3bb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj"
Jun 15 06:25:04.632810 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.632810    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7472acfb-8091-4231-b490-6e885e3fc3bb-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-dkncj\" (UID: \"7472acfb-8091-4231-b490-6e885e3fc3bb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj"
Jun 15 06:25:04.633032 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.632866    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7472acfb-8091-4231-b490-6e885e3fc3bb-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-dkncj\" (UID: \"7472acfb-8091-4231-b490-6e885e3fc3bb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj"
Jun 15 06:25:04.633032 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.632920    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7472acfb-8091-4231-b490-6e885e3fc3bb-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-dkncj\" (UID: \"7472acfb-8091-4231-b490-6e885e3fc3bb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj"
Jun 15 06:25:04.633032 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.632940    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7472acfb-8091-4231-b490-6e885e3fc3bb-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-dkncj\" (UID: \"7472acfb-8091-4231-b490-6e885e3fc3bb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj"
Jun 15 06:25:04.734493 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.734400    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7472acfb-8091-4231-b490-6e885e3fc3bb-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-dkncj\" (UID: \"7472acfb-8091-4231-b490-6e885e3fc3bb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj"
Jun 15 06:25:04.734493 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.734489    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7472acfb-8091-4231-b490-6e885e3fc3bb-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-dkncj\" (UID: \"7472acfb-8091-4231-b490-6e885e3fc3bb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj"
Jun 15 06:25:04.734726 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.734521    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7472acfb-8091-4231-b490-6e885e3fc3bb-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-dkncj\" (UID: \"7472acfb-8091-4231-b490-6e885e3fc3bb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj"
Jun 15 06:25:04.734726 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.734587    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbkxz\" (UniqueName: \"kubernetes.io/projected/7472acfb-8091-4231-b490-6e885e3fc3bb-kube-api-access-tbkxz\") pod \"isvc-predictive-xgboost-predictor-695f978978-dkncj\" (UID: \"7472acfb-8091-4231-b490-6e885e3fc3bb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj"
Jun 15 06:25:04.734726 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.734610    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7472acfb-8091-4231-b490-6e885e3fc3bb-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-dkncj\" (UID: \"7472acfb-8091-4231-b490-6e885e3fc3bb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj"
Jun 15 06:25:04.734974 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.734951    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7472acfb-8091-4231-b490-6e885e3fc3bb-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-dkncj\" (UID: \"7472acfb-8091-4231-b490-6e885e3fc3bb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj"
Jun 15 06:25:04.735247 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.735226    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7472acfb-8091-4231-b490-6e885e3fc3bb-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-dkncj\" (UID: \"7472acfb-8091-4231-b490-6e885e3fc3bb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj"
Jun 15 06:25:04.735295 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.735226    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7472acfb-8091-4231-b490-6e885e3fc3bb-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-dkncj\" (UID: \"7472acfb-8091-4231-b490-6e885e3fc3bb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj"
Jun 15 06:25:04.736963 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.736935    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7472acfb-8091-4231-b490-6e885e3fc3bb-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-dkncj\" (UID: \"7472acfb-8091-4231-b490-6e885e3fc3bb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj"
Jun 15 06:25:04.744367 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.744331    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbkxz\" (UniqueName: \"kubernetes.io/projected/7472acfb-8091-4231-b490-6e885e3fc3bb-kube-api-access-tbkxz\") pod \"isvc-predictive-xgboost-predictor-695f978978-dkncj\" (UID: \"7472acfb-8091-4231-b490-6e885e3fc3bb\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj"
Jun 15 06:25:04.828094 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.828061    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj"
Jun 15 06:25:04.961089 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:04.961056    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj"]
Jun 15 06:25:04.964346 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:25:04.964320    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7472acfb_8091_4231_b490_6e885e3fc3bb.slice/crio-784dec64fbfa6239e74f1b7b4ef065751659e0dcb581dbe794de9a4c65af1133 WatchSource:0}: Error finding container 784dec64fbfa6239e74f1b7b4ef065751659e0dcb581dbe794de9a4c65af1133: Status 404 returned error can't find the container with id 784dec64fbfa6239e74f1b7b4ef065751659e0dcb581dbe794de9a4c65af1133
Jun 15 06:25:05.060659 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:05.060622    2572 generic.go:356] "Generic (PLEG): container finished" podID="44a313fb-c1e6-4df8-b000-ea83697f6431" containerID="85ecbaf99260424ce7876e5a3c9896cf7c85f4fc07c43a6253f34e407d1f30de" exitCode=2
Jun 15 06:25:05.060823 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:05.060701    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v" event={"ID":"44a313fb-c1e6-4df8-b000-ea83697f6431","Type":"ContainerDied","Data":"85ecbaf99260424ce7876e5a3c9896cf7c85f4fc07c43a6253f34e407d1f30de"}
Jun 15 06:25:05.063186 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:05.062400    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj" event={"ID":"7472acfb-8091-4231-b490-6e885e3fc3bb","Type":"ContainerStarted","Data":"c0ed8d8157f3b20e5160492474048a9984d017e3499f61a664bf398273d6a431"}
Jun 15 06:25:05.063186 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:05.062461    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj" event={"ID":"7472acfb-8091-4231-b490-6e885e3fc3bb","Type":"ContainerStarted","Data":"784dec64fbfa6239e74f1b7b4ef065751659e0dcb581dbe794de9a4c65af1133"}
Jun 15 06:25:05.685413 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:05.685372    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v" podUID="44a313fb-c1e6-4df8-b000-ea83697f6431" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Jun 15 06:25:09.083553 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:09.083453    2572 generic.go:356] "Generic (PLEG): container finished" podID="7472acfb-8091-4231-b490-6e885e3fc3bb" containerID="c0ed8d8157f3b20e5160492474048a9984d017e3499f61a664bf398273d6a431" exitCode=0
Jun 15 06:25:09.083989 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:09.083550    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj" event={"ID":"7472acfb-8091-4231-b490-6e885e3fc3bb","Type":"ContainerDied","Data":"c0ed8d8157f3b20e5160492474048a9984d017e3499f61a664bf398273d6a431"}
Jun 15 06:25:09.452290 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:09.452265    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v"
Jun 15 06:25:09.580629 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:09.580593    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44a313fb-c1e6-4df8-b000-ea83697f6431-proxy-tls\") pod \"44a313fb-c1e6-4df8-b000-ea83697f6431\" (UID: \"44a313fb-c1e6-4df8-b000-ea83697f6431\") "
Jun 15 06:25:09.580816 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:09.580655    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44a313fb-c1e6-4df8-b000-ea83697f6431-cabundle-cert\") pod \"44a313fb-c1e6-4df8-b000-ea83697f6431\" (UID: \"44a313fb-c1e6-4df8-b000-ea83697f6431\") "
Jun 15 06:25:09.580816 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:09.580673    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44a313fb-c1e6-4df8-b000-ea83697f6431-kube-rbac-proxy-sar-config\") pod \"44a313fb-c1e6-4df8-b000-ea83697f6431\" (UID: \"44a313fb-c1e6-4df8-b000-ea83697f6431\") "
Jun 15 06:25:09.580816 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:09.580693    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgj7b\" (UniqueName: \"kubernetes.io/projected/44a313fb-c1e6-4df8-b000-ea83697f6431-kube-api-access-lgj7b\") pod \"44a313fb-c1e6-4df8-b000-ea83697f6431\" (UID: \"44a313fb-c1e6-4df8-b000-ea83697f6431\") "
Jun 15 06:25:09.580816 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:09.580735    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44a313fb-c1e6-4df8-b000-ea83697f6431-kserve-provision-location\") pod \"44a313fb-c1e6-4df8-b000-ea83697f6431\" (UID: \"44a313fb-c1e6-4df8-b000-ea83697f6431\") "
Jun 15 06:25:09.581159 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:09.581134    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44a313fb-c1e6-4df8-b000-ea83697f6431-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "44a313fb-c1e6-4df8-b000-ea83697f6431" (UID: "44a313fb-c1e6-4df8-b000-ea83697f6431"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:25:09.581159 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:09.581146    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a313fb-c1e6-4df8-b000-ea83697f6431-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "44a313fb-c1e6-4df8-b000-ea83697f6431" (UID: "44a313fb-c1e6-4df8-b000-ea83697f6431"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:25:09.581269 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:09.581130    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44a313fb-c1e6-4df8-b000-ea83697f6431-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "44a313fb-c1e6-4df8-b000-ea83697f6431" (UID: "44a313fb-c1e6-4df8-b000-ea83697f6431"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:25:09.582846 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:09.582827    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44a313fb-c1e6-4df8-b000-ea83697f6431-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "44a313fb-c1e6-4df8-b000-ea83697f6431" (UID: "44a313fb-c1e6-4df8-b000-ea83697f6431"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:25:09.582846 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:09.582837    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a313fb-c1e6-4df8-b000-ea83697f6431-kube-api-access-lgj7b" (OuterVolumeSpecName: "kube-api-access-lgj7b") pod "44a313fb-c1e6-4df8-b000-ea83697f6431" (UID: "44a313fb-c1e6-4df8-b000-ea83697f6431"). InnerVolumeSpecName "kube-api-access-lgj7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:25:09.681943 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:09.681906    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/44a313fb-c1e6-4df8-b000-ea83697f6431-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:25:09.681943 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:09.681936    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/44a313fb-c1e6-4df8-b000-ea83697f6431-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:25:09.681943 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:09.681946    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/44a313fb-c1e6-4df8-b000-ea83697f6431-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:25:09.681943 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:09.681954    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/44a313fb-c1e6-4df8-b000-ea83697f6431-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:25:09.682222 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:09.681962    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgj7b\" (UniqueName: \"kubernetes.io/projected/44a313fb-c1e6-4df8-b000-ea83697f6431-kube-api-access-lgj7b\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:25:10.090373 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:10.090264    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj" event={"ID":"7472acfb-8091-4231-b490-6e885e3fc3bb","Type":"ContainerStarted","Data":"a8e77d30c886cece8c761e271042db17ff24c5935addf67f0e35d983b4d21d4e"}
Jun 15 06:25:10.090373 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:10.090342    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj" event={"ID":"7472acfb-8091-4231-b490-6e885e3fc3bb","Type":"ContainerStarted","Data":"4cdc7c8234887a1f4a12ca8651b0c2b58f035f3cb6ef426fefe1d28a276b74c0"}
Jun 15 06:25:10.090841 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:10.090656    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj"
Jun 15 06:25:10.090841 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:10.090807    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj"
Jun 15 06:25:10.092177 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:10.092155    2572 generic.go:356] "Generic (PLEG): container finished" podID="44a313fb-c1e6-4df8-b000-ea83697f6431" containerID="c46e46a11ce12d6fa61680a0c8f9b1ba2954767a6f6b9a27796aaedd5521d1c7" exitCode=0
Jun 15 06:25:10.092296 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:10.092210    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v"
Jun 15 06:25:10.092296 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:10.092222    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v" event={"ID":"44a313fb-c1e6-4df8-b000-ea83697f6431","Type":"ContainerDied","Data":"c46e46a11ce12d6fa61680a0c8f9b1ba2954767a6f6b9a27796aaedd5521d1c7"}
Jun 15 06:25:10.092296 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:10.092269    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v" event={"ID":"44a313fb-c1e6-4df8-b000-ea83697f6431","Type":"ContainerDied","Data":"180a57667fe7757127079804e371c2de06140c132a3241c8a17d32fac23adeab"}
Jun 15 06:25:10.092296 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:10.092271    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj" podUID="7472acfb-8091-4231-b490-6e885e3fc3bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jun 15 06:25:10.092459 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:10.092300    2572 scope.go:117] "RemoveContainer" containerID="85ecbaf99260424ce7876e5a3c9896cf7c85f4fc07c43a6253f34e407d1f30de"
Jun 15 06:25:10.101208 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:10.101191    2572 scope.go:117] "RemoveContainer" containerID="c46e46a11ce12d6fa61680a0c8f9b1ba2954767a6f6b9a27796aaedd5521d1c7"
Jun 15 06:25:10.108455 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:10.108435    2572 scope.go:117] "RemoveContainer" containerID="f24179180e56a7c1cbc71873fa648a38787dd2450d07aed108fe88c858daedd8"
Jun 15 06:25:10.112369 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:10.112310    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj" podStartSLOduration=2.335439741 podStartE2EDuration="6.112295802s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.776856061s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:25:10.11097533 +0000 UTC m=+2225.259155332" watchObservedRunningTime="2026-06-15 06:25:10.112295802 +0000 UTC m=+2225.260475872"
Jun 15 06:25:10.115698 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:10.115683    2572 scope.go:117] "RemoveContainer" containerID="85ecbaf99260424ce7876e5a3c9896cf7c85f4fc07c43a6253f34e407d1f30de"
Jun 15 06:25:10.115933 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:25:10.115916    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85ecbaf99260424ce7876e5a3c9896cf7c85f4fc07c43a6253f34e407d1f30de\": container with ID starting with 85ecbaf99260424ce7876e5a3c9896cf7c85f4fc07c43a6253f34e407d1f30de not found: ID does not exist" containerID="85ecbaf99260424ce7876e5a3c9896cf7c85f4fc07c43a6253f34e407d1f30de"
Jun 15 06:25:10.115994 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:10.115939    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85ecbaf99260424ce7876e5a3c9896cf7c85f4fc07c43a6253f34e407d1f30de"} err="failed to get container status \"85ecbaf99260424ce7876e5a3c9896cf7c85f4fc07c43a6253f34e407d1f30de\": rpc error: code = NotFound desc = could not find container \"85ecbaf99260424ce7876e5a3c9896cf7c85f4fc07c43a6253f34e407d1f30de\": container with ID starting with 85ecbaf99260424ce7876e5a3c9896cf7c85f4fc07c43a6253f34e407d1f30de not found: ID does not exist"
Jun 15 06:25:10.115994 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:10.115953    2572 scope.go:117] "RemoveContainer" containerID="c46e46a11ce12d6fa61680a0c8f9b1ba2954767a6f6b9a27796aaedd5521d1c7"
Jun 15 06:25:10.116173 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:25:10.116156    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c46e46a11ce12d6fa61680a0c8f9b1ba2954767a6f6b9a27796aaedd5521d1c7\": container with ID starting with c46e46a11ce12d6fa61680a0c8f9b1ba2954767a6f6b9a27796aaedd5521d1c7 not found: ID does not exist" containerID="c46e46a11ce12d6fa61680a0c8f9b1ba2954767a6f6b9a27796aaedd5521d1c7"
Jun 15 06:25:10.116214 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:10.116177    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c46e46a11ce12d6fa61680a0c8f9b1ba2954767a6f6b9a27796aaedd5521d1c7"} err="failed to get container status \"c46e46a11ce12d6fa61680a0c8f9b1ba2954767a6f6b9a27796aaedd5521d1c7\": rpc error: code = NotFound desc = could not find container \"c46e46a11ce12d6fa61680a0c8f9b1ba2954767a6f6b9a27796aaedd5521d1c7\": container with ID starting with c46e46a11ce12d6fa61680a0c8f9b1ba2954767a6f6b9a27796aaedd5521d1c7 not found: ID does not exist"
Jun 15 06:25:10.116214 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:10.116189    2572 scope.go:117] "RemoveContainer" containerID="f24179180e56a7c1cbc71873fa648a38787dd2450d07aed108fe88c858daedd8"
Jun 15 06:25:10.116402 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:25:10.116384    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f24179180e56a7c1cbc71873fa648a38787dd2450d07aed108fe88c858daedd8\": container with ID starting with f24179180e56a7c1cbc71873fa648a38787dd2450d07aed108fe88c858daedd8 not found: ID does not exist" containerID="f24179180e56a7c1cbc71873fa648a38787dd2450d07aed108fe88c858daedd8"
Jun 15 06:25:10.116451 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:10.116408    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f24179180e56a7c1cbc71873fa648a38787dd2450d07aed108fe88c858daedd8"} err="failed to get container status \"f24179180e56a7c1cbc71873fa648a38787dd2450d07aed108fe88c858daedd8\": rpc error: code = NotFound desc = could not find container \"f24179180e56a7c1cbc71873fa648a38787dd2450d07aed108fe88c858daedd8\": container with ID starting with f24179180e56a7c1cbc71873fa648a38787dd2450d07aed108fe88c858daedd8 not found: ID does not exist"
Jun 15 06:25:10.125659 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:10.125640    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v"]
Jun 15 06:25:10.130932 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:10.130910    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-5c69v"]
Jun 15 06:25:11.098112 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:11.098013    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj" podUID="7472acfb-8091-4231-b490-6e885e3fc3bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jun 15 06:25:11.324134 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:11.324090    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44a313fb-c1e6-4df8-b000-ea83697f6431" path="/var/lib/kubelet/pods/44a313fb-c1e6-4df8-b000-ea83697f6431/volumes"
Jun 15 06:25:16.102220 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:16.102144    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj"
Jun 15 06:25:16.102636 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:16.102611    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj" podUID="7472acfb-8091-4231-b490-6e885e3fc3bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jun 15 06:25:26.102707 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:26.102672    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj" podUID="7472acfb-8091-4231-b490-6e885e3fc3bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jun 15 06:25:36.103612 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:36.103572    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj" podUID="7472acfb-8091-4231-b490-6e885e3fc3bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jun 15 06:25:46.102576 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:46.102535    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj" podUID="7472acfb-8091-4231-b490-6e885e3fc3bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jun 15 06:25:56.103592 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:25:56.103553    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj" podUID="7472acfb-8091-4231-b490-6e885e3fc3bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jun 15 06:26:06.103317 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:06.103281    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj" podUID="7472acfb-8091-4231-b490-6e885e3fc3bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jun 15 06:26:16.102843 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:16.102804    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj" podUID="7472acfb-8091-4231-b490-6e885e3fc3bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jun 15 06:26:26.103497 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:26.103469    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj"
Jun 15 06:26:34.588550 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:34.588512    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj"]
Jun 15 06:26:34.588961 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:34.588780    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj" podUID="7472acfb-8091-4231-b490-6e885e3fc3bb" containerName="kserve-container" containerID="cri-o://4cdc7c8234887a1f4a12ca8651b0c2b58f035f3cb6ef426fefe1d28a276b74c0" gracePeriod=30
Jun 15 06:26:34.588961 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:34.588818    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj" podUID="7472acfb-8091-4231-b490-6e885e3fc3bb" containerName="kube-rbac-proxy" containerID="cri-o://a8e77d30c886cece8c761e271042db17ff24c5935addf67f0e35d983b4d21d4e" gracePeriod=30
Jun 15 06:26:35.494223 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:35.494182    2572 generic.go:356] "Generic (PLEG): container finished" podID="7472acfb-8091-4231-b490-6e885e3fc3bb" containerID="a8e77d30c886cece8c761e271042db17ff24c5935addf67f0e35d983b4d21d4e" exitCode=2
Jun 15 06:26:35.494415 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:35.494240    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj" event={"ID":"7472acfb-8091-4231-b490-6e885e3fc3bb","Type":"ContainerDied","Data":"a8e77d30c886cece8c761e271042db17ff24c5935addf67f0e35d983b4d21d4e"}
Jun 15 06:26:36.098859 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:36.098824    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj" podUID="7472acfb-8091-4231-b490-6e885e3fc3bb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Jun 15 06:26:36.103162 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:36.103137    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj" podUID="7472acfb-8091-4231-b490-6e885e3fc3bb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jun 15 06:26:39.335245 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.335221    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj"
Jun 15 06:26:39.475463 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.475350    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7472acfb-8091-4231-b490-6e885e3fc3bb-kserve-provision-location\") pod \"7472acfb-8091-4231-b490-6e885e3fc3bb\" (UID: \"7472acfb-8091-4231-b490-6e885e3fc3bb\") "
Jun 15 06:26:39.475463 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.475427    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7472acfb-8091-4231-b490-6e885e3fc3bb-cabundle-cert\") pod \"7472acfb-8091-4231-b490-6e885e3fc3bb\" (UID: \"7472acfb-8091-4231-b490-6e885e3fc3bb\") "
Jun 15 06:26:39.475690 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.475472    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7472acfb-8091-4231-b490-6e885e3fc3bb-kube-rbac-proxy-sar-config\") pod \"7472acfb-8091-4231-b490-6e885e3fc3bb\" (UID: \"7472acfb-8091-4231-b490-6e885e3fc3bb\") "
Jun 15 06:26:39.475690 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.475535    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbkxz\" (UniqueName: \"kubernetes.io/projected/7472acfb-8091-4231-b490-6e885e3fc3bb-kube-api-access-tbkxz\") pod \"7472acfb-8091-4231-b490-6e885e3fc3bb\" (UID: \"7472acfb-8091-4231-b490-6e885e3fc3bb\") "
Jun 15 06:26:39.475690 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.475595    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7472acfb-8091-4231-b490-6e885e3fc3bb-proxy-tls\") pod \"7472acfb-8091-4231-b490-6e885e3fc3bb\" (UID: \"7472acfb-8091-4231-b490-6e885e3fc3bb\") "
Jun 15 06:26:39.475848 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.475733    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7472acfb-8091-4231-b490-6e885e3fc3bb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7472acfb-8091-4231-b490-6e885e3fc3bb" (UID: "7472acfb-8091-4231-b490-6e885e3fc3bb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:26:39.475848 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.475789    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7472acfb-8091-4231-b490-6e885e3fc3bb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7472acfb-8091-4231-b490-6e885e3fc3bb" (UID: "7472acfb-8091-4231-b490-6e885e3fc3bb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:26:39.475963 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.475921    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7472acfb-8091-4231-b490-6e885e3fc3bb-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:26:39.475963 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.475940    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7472acfb-8091-4231-b490-6e885e3fc3bb-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:26:39.475963 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.475938    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7472acfb-8091-4231-b490-6e885e3fc3bb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7472acfb-8091-4231-b490-6e885e3fc3bb" (UID: "7472acfb-8091-4231-b490-6e885e3fc3bb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:26:39.477680 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.477654    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7472acfb-8091-4231-b490-6e885e3fc3bb-kube-api-access-tbkxz" (OuterVolumeSpecName: "kube-api-access-tbkxz") pod "7472acfb-8091-4231-b490-6e885e3fc3bb" (UID: "7472acfb-8091-4231-b490-6e885e3fc3bb"). InnerVolumeSpecName "kube-api-access-tbkxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:26:39.477789 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.477686    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7472acfb-8091-4231-b490-6e885e3fc3bb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7472acfb-8091-4231-b490-6e885e3fc3bb" (UID: "7472acfb-8091-4231-b490-6e885e3fc3bb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:26:39.516898 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.516868    2572 generic.go:356] "Generic (PLEG): container finished" podID="7472acfb-8091-4231-b490-6e885e3fc3bb" containerID="4cdc7c8234887a1f4a12ca8651b0c2b58f035f3cb6ef426fefe1d28a276b74c0" exitCode=0
Jun 15 06:26:39.517078 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.516921    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj"
Jun 15 06:26:39.517078 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.516951    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj" event={"ID":"7472acfb-8091-4231-b490-6e885e3fc3bb","Type":"ContainerDied","Data":"4cdc7c8234887a1f4a12ca8651b0c2b58f035f3cb6ef426fefe1d28a276b74c0"}
Jun 15 06:26:39.517078 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.517010    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj" event={"ID":"7472acfb-8091-4231-b490-6e885e3fc3bb","Type":"ContainerDied","Data":"784dec64fbfa6239e74f1b7b4ef065751659e0dcb581dbe794de9a4c65af1133"}
Jun 15 06:26:39.517078 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.517035    2572 scope.go:117] "RemoveContainer" containerID="a8e77d30c886cece8c761e271042db17ff24c5935addf67f0e35d983b4d21d4e"
Jun 15 06:26:39.525671 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.525655    2572 scope.go:117] "RemoveContainer" containerID="4cdc7c8234887a1f4a12ca8651b0c2b58f035f3cb6ef426fefe1d28a276b74c0"
Jun 15 06:26:39.533201 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.533185    2572 scope.go:117] "RemoveContainer" containerID="c0ed8d8157f3b20e5160492474048a9984d017e3499f61a664bf398273d6a431"
Jun 15 06:26:39.540765 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.540748    2572 scope.go:117] "RemoveContainer" containerID="a8e77d30c886cece8c761e271042db17ff24c5935addf67f0e35d983b4d21d4e"
Jun 15 06:26:39.541022 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:26:39.541001    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8e77d30c886cece8c761e271042db17ff24c5935addf67f0e35d983b4d21d4e\": container with ID starting with a8e77d30c886cece8c761e271042db17ff24c5935addf67f0e35d983b4d21d4e not found: ID does not exist" containerID="a8e77d30c886cece8c761e271042db17ff24c5935addf67f0e35d983b4d21d4e"
Jun 15 06:26:39.541068 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.541032    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8e77d30c886cece8c761e271042db17ff24c5935addf67f0e35d983b4d21d4e"} err="failed to get container status \"a8e77d30c886cece8c761e271042db17ff24c5935addf67f0e35d983b4d21d4e\": rpc error: code = NotFound desc = could not find container \"a8e77d30c886cece8c761e271042db17ff24c5935addf67f0e35d983b4d21d4e\": container with ID starting with a8e77d30c886cece8c761e271042db17ff24c5935addf67f0e35d983b4d21d4e not found: ID does not exist"
Jun 15 06:26:39.541068 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.541050    2572 scope.go:117] "RemoveContainer" containerID="4cdc7c8234887a1f4a12ca8651b0c2b58f035f3cb6ef426fefe1d28a276b74c0"
Jun 15 06:26:39.541320 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:26:39.541299    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cdc7c8234887a1f4a12ca8651b0c2b58f035f3cb6ef426fefe1d28a276b74c0\": container with ID starting with 4cdc7c8234887a1f4a12ca8651b0c2b58f035f3cb6ef426fefe1d28a276b74c0 not found: ID does not exist" containerID="4cdc7c8234887a1f4a12ca8651b0c2b58f035f3cb6ef426fefe1d28a276b74c0"
Jun 15 06:26:39.541417 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.541329    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cdc7c8234887a1f4a12ca8651b0c2b58f035f3cb6ef426fefe1d28a276b74c0"} err="failed to get container status \"4cdc7c8234887a1f4a12ca8651b0c2b58f035f3cb6ef426fefe1d28a276b74c0\": rpc error: code = NotFound desc = could not find container \"4cdc7c8234887a1f4a12ca8651b0c2b58f035f3cb6ef426fefe1d28a276b74c0\": container with ID starting with 4cdc7c8234887a1f4a12ca8651b0c2b58f035f3cb6ef426fefe1d28a276b74c0 not found: ID does not exist"
Jun 15 06:26:39.541417 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.541349    2572 scope.go:117] "RemoveContainer" containerID="c0ed8d8157f3b20e5160492474048a9984d017e3499f61a664bf398273d6a431"
Jun 15 06:26:39.541666 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:26:39.541646    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0ed8d8157f3b20e5160492474048a9984d017e3499f61a664bf398273d6a431\": container with ID starting with c0ed8d8157f3b20e5160492474048a9984d017e3499f61a664bf398273d6a431 not found: ID does not exist" containerID="c0ed8d8157f3b20e5160492474048a9984d017e3499f61a664bf398273d6a431"
Jun 15 06:26:39.541798 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.541668    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0ed8d8157f3b20e5160492474048a9984d017e3499f61a664bf398273d6a431"} err="failed to get container status \"c0ed8d8157f3b20e5160492474048a9984d017e3499f61a664bf398273d6a431\": rpc error: code = NotFound desc = could not find container \"c0ed8d8157f3b20e5160492474048a9984d017e3499f61a664bf398273d6a431\": container with ID starting with c0ed8d8157f3b20e5160492474048a9984d017e3499f61a664bf398273d6a431 not found: ID does not exist"
Jun 15 06:26:39.542282 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.542265    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj"]
Jun 15 06:26:39.546657 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.546637    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-dkncj"]
Jun 15 06:26:39.576778 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.576751    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7472acfb-8091-4231-b490-6e885e3fc3bb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:26:39.576778 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.576778    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbkxz\" (UniqueName: \"kubernetes.io/projected/7472acfb-8091-4231-b490-6e885e3fc3bb-kube-api-access-tbkxz\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:26:39.576950 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:39.576789    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7472acfb-8091-4231-b490-6e885e3fc3bb-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:26:41.323322 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:26:41.323279    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7472acfb-8091-4231-b490-6e885e3fc3bb" path="/var/lib/kubelet/pods/7472acfb-8091-4231-b490-6e885e3fc3bb/volumes"
Jun 15 06:28:05.483897 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:28:05.483865    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 06:28:05.500640 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:28:05.500616    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 06:33:05.510546 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:05.510519    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 06:33:05.527767 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:05.527745    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 06:33:07.913615 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:07.913563    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq"]
Jun 15 06:33:07.914034 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:07.913968    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7472acfb-8091-4231-b490-6e885e3fc3bb" containerName="kserve-container"
Jun 15 06:33:07.914034 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:07.913986    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7472acfb-8091-4231-b490-6e885e3fc3bb" containerName="kserve-container"
Jun 15 06:33:07.914034 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:07.914015    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44a313fb-c1e6-4df8-b000-ea83697f6431" containerName="storage-initializer"
Jun 15 06:33:07.914034 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:07.914023    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="44a313fb-c1e6-4df8-b000-ea83697f6431" containerName="storage-initializer"
Jun 15 06:33:07.914241 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:07.914040    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7472acfb-8091-4231-b490-6e885e3fc3bb" containerName="storage-initializer"
Jun 15 06:33:07.914241 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:07.914049    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7472acfb-8091-4231-b490-6e885e3fc3bb" containerName="storage-initializer"
Jun 15 06:33:07.914241 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:07.914071    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44a313fb-c1e6-4df8-b000-ea83697f6431" containerName="kserve-container"
Jun 15 06:33:07.914241 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:07.914079    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="44a313fb-c1e6-4df8-b000-ea83697f6431" containerName="kserve-container"
Jun 15 06:33:07.914241 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:07.914087    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44a313fb-c1e6-4df8-b000-ea83697f6431" containerName="kube-rbac-proxy"
Jun 15 06:33:07.914241 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:07.914094    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="44a313fb-c1e6-4df8-b000-ea83697f6431" containerName="kube-rbac-proxy"
Jun 15 06:33:07.914241 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:07.914103    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7472acfb-8091-4231-b490-6e885e3fc3bb" containerName="kube-rbac-proxy"
Jun 15 06:33:07.914241 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:07.914118    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="7472acfb-8091-4231-b490-6e885e3fc3bb" containerName="kube-rbac-proxy"
Jun 15 06:33:07.914241 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:07.914197    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="44a313fb-c1e6-4df8-b000-ea83697f6431" containerName="kserve-container"
Jun 15 06:33:07.914241 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:07.914210    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="44a313fb-c1e6-4df8-b000-ea83697f6431" containerName="kube-rbac-proxy"
Jun 15 06:33:07.914241 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:07.914224    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7472acfb-8091-4231-b490-6e885e3fc3bb" containerName="kube-rbac-proxy"
Jun 15 06:33:07.914241 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:07.914234    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="7472acfb-8091-4231-b490-6e885e3fc3bb" containerName="kserve-container"
Jun 15 06:33:07.917462 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:07.917442    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq"
Jun 15 06:33:07.919560 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:07.919539    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jun 15 06:33:07.919780 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:07.919756    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-predictor-serving-cert\""
Jun 15 06:33:07.919780 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:07.919777    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-q7vt7\""
Jun 15 06:33:07.919958 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:07.919784    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 15 06:33:07.919958 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:07.919811    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-kube-rbac-proxy-sar-config\""
Jun 15 06:33:07.920409 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:07.920389    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 15 06:33:07.920468 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:07.920418    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jun 15 06:33:07.931569 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:07.931542    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq"]
Jun 15 06:33:08.010214 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:08.010182    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5bdbc4b78f-qh4pq\" (UID: \"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq"
Jun 15 06:33:08.010420 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:08.010238    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-kserve-provision-location\") pod \"isvc-sklearn-predictor-5bdbc4b78f-qh4pq\" (UID: \"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq"
Jun 15 06:33:08.010420 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:08.010301    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-proxy-tls\") pod \"isvc-sklearn-predictor-5bdbc4b78f-qh4pq\" (UID: \"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq"
Jun 15 06:33:08.010527 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:08.010411    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-cabundle-cert\") pod \"isvc-sklearn-predictor-5bdbc4b78f-qh4pq\" (UID: \"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq"
Jun 15 06:33:08.010527 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:08.010445    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msvtf\" (UniqueName: \"kubernetes.io/projected/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-kube-api-access-msvtf\") pod \"isvc-sklearn-predictor-5bdbc4b78f-qh4pq\" (UID: \"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq"
Jun 15 06:33:08.111286 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:08.111256    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-cabundle-cert\") pod \"isvc-sklearn-predictor-5bdbc4b78f-qh4pq\" (UID: \"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq"
Jun 15 06:33:08.111286 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:08.111289    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msvtf\" (UniqueName: \"kubernetes.io/projected/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-kube-api-access-msvtf\") pod \"isvc-sklearn-predictor-5bdbc4b78f-qh4pq\" (UID: \"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq"
Jun 15 06:33:08.111515 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:08.111310    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5bdbc4b78f-qh4pq\" (UID: \"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq"
Jun 15 06:33:08.111576 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:08.111537    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-kserve-provision-location\") pod \"isvc-sklearn-predictor-5bdbc4b78f-qh4pq\" (UID: \"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq"
Jun 15 06:33:08.111646 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:08.111631    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-proxy-tls\") pod \"isvc-sklearn-predictor-5bdbc4b78f-qh4pq\" (UID: \"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq"
Jun 15 06:33:08.111930 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:08.111910    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-kserve-provision-location\") pod \"isvc-sklearn-predictor-5bdbc4b78f-qh4pq\" (UID: \"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq"
Jun 15 06:33:08.111970 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:08.111929    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-predictor-5bdbc4b78f-qh4pq\" (UID: \"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq"
Jun 15 06:33:08.112024 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:08.112008    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-cabundle-cert\") pod \"isvc-sklearn-predictor-5bdbc4b78f-qh4pq\" (UID: \"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq"
Jun 15 06:33:08.114069 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:08.114051    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-proxy-tls\") pod \"isvc-sklearn-predictor-5bdbc4b78f-qh4pq\" (UID: \"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq"
Jun 15 06:33:08.120519 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:08.120500    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msvtf\" (UniqueName: \"kubernetes.io/projected/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-kube-api-access-msvtf\") pod \"isvc-sklearn-predictor-5bdbc4b78f-qh4pq\" (UID: \"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6\") " pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq"
Jun 15 06:33:08.233599 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:08.233515    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq"
Jun 15 06:33:08.370649 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:08.370623    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq"]
Jun 15 06:33:08.372709 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:33:08.372680    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e8d87fd_fa3c_41ca_bf83_f0abd41a81e6.slice/crio-c63115cb837924e8a1aa649c38f1488efbcc6a71a71f4d347c1fcb557fa4ef93 WatchSource:0}: Error finding container c63115cb837924e8a1aa649c38f1488efbcc6a71a71f4d347c1fcb557fa4ef93: Status 404 returned error can't find the container with id c63115cb837924e8a1aa649c38f1488efbcc6a71a71f4d347c1fcb557fa4ef93
Jun 15 06:33:09.181385 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:09.181336    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq" event={"ID":"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6","Type":"ContainerStarted","Data":"bc2bb3e9285e636eadbd009bc0ce8d928e85edaae2c8f67994bd1a651f8c10c2"}
Jun 15 06:33:09.181775 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:09.181408    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq" event={"ID":"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6","Type":"ContainerStarted","Data":"c63115cb837924e8a1aa649c38f1488efbcc6a71a71f4d347c1fcb557fa4ef93"}
Jun 15 06:33:13.199847 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:13.199809    2572 generic.go:356] "Generic (PLEG): container finished" podID="3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" containerID="bc2bb3e9285e636eadbd009bc0ce8d928e85edaae2c8f67994bd1a651f8c10c2" exitCode=0
Jun 15 06:33:13.200254 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:13.199869    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq" event={"ID":"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6","Type":"ContainerDied","Data":"bc2bb3e9285e636eadbd009bc0ce8d928e85edaae2c8f67994bd1a651f8c10c2"}
Jun 15 06:33:14.206470 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:14.206434    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq" event={"ID":"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6","Type":"ContainerStarted","Data":"c128171f8ffbf0546e318032dd25558bfe40bc23681465c4c72cc504dc47709f"}
Jun 15 06:33:14.206855 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:14.206495    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq" event={"ID":"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6","Type":"ContainerStarted","Data":"ddba48d270c31b7a90cf1473c98f8bdd2d9470a15b7d23a9ec2f199485c4ce46"}
Jun 15 06:33:14.206855 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:14.206681    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq"
Jun 15 06:33:14.226538 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:14.226489    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq" podStartSLOduration=3.321128409 podStartE2EDuration="7.22647339s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.905344981s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:33:14.225264201 +0000 UTC m=+2709.373444195" watchObservedRunningTime="2026-06-15 06:33:14.22647339 +0000 UTC m=+2709.374653386"
Jun 15 06:33:15.211494 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:15.211457    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq"
Jun 15 06:33:15.212816 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:15.212786    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq" podUID="3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jun 15 06:33:16.215966 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:16.215923    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq" podUID="3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jun 15 06:33:21.221755 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:21.221726    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq"
Jun 15 06:33:21.222238 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:21.222209    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq" podUID="3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jun 15 06:33:31.222187 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:31.222145    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq" podUID="3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jun 15 06:33:41.222735 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:41.222695    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq" podUID="3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jun 15 06:33:51.222671 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:33:51.222631    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq" podUID="3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jun 15 06:34:01.223105 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:01.223058    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq" podUID="3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jun 15 06:34:11.223103 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:11.223064    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq" podUID="3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jun 15 06:34:21.222945 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:21.222859    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq" podUID="3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jun 15 06:34:31.222512 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:31.222479    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq"
Jun 15 06:34:38.076453 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:38.076419    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq"]
Jun 15 06:34:38.076931 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:38.076879    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq" podUID="3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" containerName="kserve-container" containerID="cri-o://ddba48d270c31b7a90cf1473c98f8bdd2d9470a15b7d23a9ec2f199485c4ce46" gracePeriod=30
Jun 15 06:34:38.077003 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:38.076934    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq" podUID="3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" containerName="kube-rbac-proxy" containerID="cri-o://c128171f8ffbf0546e318032dd25558bfe40bc23681465c4c72cc504dc47709f" gracePeriod=30
Jun 15 06:34:38.176046 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:38.176010    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms"]
Jun 15 06:34:38.179574 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:38.179553    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms"
Jun 15 06:34:38.181656 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:38.181630    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-kube-rbac-proxy-sar-config\""
Jun 15 06:34:38.181851 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:38.181837    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sklearn-v2-mlserver-predictor-serving-cert\""
Jun 15 06:34:38.190693 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:38.190671    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms"]
Jun 15 06:34:38.216272 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:38.216242    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/add34785-94c3-4bab-aec9-f9257d5b443e-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms\" (UID: \"add34785-94c3-4bab-aec9-f9257d5b443e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms"
Jun 15 06:34:38.216488 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:38.216283    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/add34785-94c3-4bab-aec9-f9257d5b443e-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms\" (UID: \"add34785-94c3-4bab-aec9-f9257d5b443e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms"
Jun 15 06:34:38.216488 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:38.216378    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/add34785-94c3-4bab-aec9-f9257d5b443e-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms\" (UID: \"add34785-94c3-4bab-aec9-f9257d5b443e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms"
Jun 15 06:34:38.216488 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:38.216428    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqm2b\" (UniqueName: \"kubernetes.io/projected/add34785-94c3-4bab-aec9-f9257d5b443e-kube-api-access-zqm2b\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms\" (UID: \"add34785-94c3-4bab-aec9-f9257d5b443e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms"
Jun 15 06:34:38.216488 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:38.216469    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/add34785-94c3-4bab-aec9-f9257d5b443e-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms\" (UID: \"add34785-94c3-4bab-aec9-f9257d5b443e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms"
Jun 15 06:34:38.317486 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:38.317449    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/add34785-94c3-4bab-aec9-f9257d5b443e-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms\" (UID: \"add34785-94c3-4bab-aec9-f9257d5b443e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms"
Jun 15 06:34:38.317670 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:38.317500    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqm2b\" (UniqueName: \"kubernetes.io/projected/add34785-94c3-4bab-aec9-f9257d5b443e-kube-api-access-zqm2b\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms\" (UID: \"add34785-94c3-4bab-aec9-f9257d5b443e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms"
Jun 15 06:34:38.317670 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:38.317539    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/add34785-94c3-4bab-aec9-f9257d5b443e-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms\" (UID: \"add34785-94c3-4bab-aec9-f9257d5b443e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms"
Jun 15 06:34:38.317670 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:38.317567    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/add34785-94c3-4bab-aec9-f9257d5b443e-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms\" (UID: \"add34785-94c3-4bab-aec9-f9257d5b443e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms"
Jun 15 06:34:38.317850 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:38.317714    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/add34785-94c3-4bab-aec9-f9257d5b443e-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms\" (UID: \"add34785-94c3-4bab-aec9-f9257d5b443e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms"
Jun 15 06:34:38.317978 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:38.317959    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/add34785-94c3-4bab-aec9-f9257d5b443e-kserve-provision-location\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms\" (UID: \"add34785-94c3-4bab-aec9-f9257d5b443e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms"
Jun 15 06:34:38.318153 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:38.318134    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/add34785-94c3-4bab-aec9-f9257d5b443e-cabundle-cert\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms\" (UID: \"add34785-94c3-4bab-aec9-f9257d5b443e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms"
Jun 15 06:34:38.318250 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:38.318231    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/add34785-94c3-4bab-aec9-f9257d5b443e-kube-rbac-proxy-sar-config\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms\" (UID: \"add34785-94c3-4bab-aec9-f9257d5b443e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms"
Jun 15 06:34:38.320105 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:38.320086    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/add34785-94c3-4bab-aec9-f9257d5b443e-proxy-tls\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms\" (UID: \"add34785-94c3-4bab-aec9-f9257d5b443e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms"
Jun 15 06:34:38.326824 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:38.326775    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqm2b\" (UniqueName: \"kubernetes.io/projected/add34785-94c3-4bab-aec9-f9257d5b443e-kube-api-access-zqm2b\") pod \"sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms\" (UID: \"add34785-94c3-4bab-aec9-f9257d5b443e\") " pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms"
Jun 15 06:34:38.491633 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:38.491600    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms"
Jun 15 06:34:38.592912 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:38.592789    2572 generic.go:356] "Generic (PLEG): container finished" podID="3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" containerID="c128171f8ffbf0546e318032dd25558bfe40bc23681465c4c72cc504dc47709f" exitCode=2
Jun 15 06:34:38.592912 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:38.592884    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq" event={"ID":"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6","Type":"ContainerDied","Data":"c128171f8ffbf0546e318032dd25558bfe40bc23681465c4c72cc504dc47709f"}
Jun 15 06:34:38.622849 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:38.622818    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms"]
Jun 15 06:34:38.625760 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:34:38.625738    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd34785_94c3_4bab_aec9_f9257d5b443e.slice/crio-dde7c2493b4cece61a23a668eb6a040b176937560a5b6c95c2762f0c17703dd2 WatchSource:0}: Error finding container dde7c2493b4cece61a23a668eb6a040b176937560a5b6c95c2762f0c17703dd2: Status 404 returned error can't find the container with id dde7c2493b4cece61a23a668eb6a040b176937560a5b6c95c2762f0c17703dd2
Jun 15 06:34:39.599925 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:39.599875    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms" event={"ID":"add34785-94c3-4bab-aec9-f9257d5b443e","Type":"ContainerStarted","Data":"b3504b509e91366f24abc0471e0c9ededc804d5ed487be731e5a97bb4d291117"}
Jun 15 06:34:39.600346 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:39.599945    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms" event={"ID":"add34785-94c3-4bab-aec9-f9257d5b443e","Type":"ContainerStarted","Data":"dde7c2493b4cece61a23a668eb6a040b176937560a5b6c95c2762f0c17703dd2"}
Jun 15 06:34:41.216413 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:41.216348    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq" podUID="3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Jun 15 06:34:41.223107 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:41.223080    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq" podUID="3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jun 15 06:34:42.615785 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:42.615746    2572 generic.go:356] "Generic (PLEG): container finished" podID="add34785-94c3-4bab-aec9-f9257d5b443e" containerID="b3504b509e91366f24abc0471e0c9ededc804d5ed487be731e5a97bb4d291117" exitCode=0
Jun 15 06:34:42.616123 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:42.615815    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms" event={"ID":"add34785-94c3-4bab-aec9-f9257d5b443e","Type":"ContainerDied","Data":"b3504b509e91366f24abc0471e0c9ededc804d5ed487be731e5a97bb4d291117"}
Jun 15 06:34:42.822287 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:42.822266    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq"
Jun 15 06:34:42.855309 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:42.855276    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-kserve-provision-location\") pod \"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6\" (UID: \"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6\") "
Jun 15 06:34:42.855501 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:42.855316    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-cabundle-cert\") pod \"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6\" (UID: \"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6\") "
Jun 15 06:34:42.855501 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:42.855404    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msvtf\" (UniqueName: \"kubernetes.io/projected/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-kube-api-access-msvtf\") pod \"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6\" (UID: \"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6\") "
Jun 15 06:34:42.855501 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:42.855454    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-proxy-tls\") pod \"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6\" (UID: \"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6\") "
Jun 15 06:34:42.855501 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:42.855486    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-kube-rbac-proxy-sar-config\") pod \"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6\" (UID: \"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6\") "
Jun 15 06:34:42.855711 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:42.855640    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" (UID: "3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:34:42.855759 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:42.855733    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" (UID: "3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:34:42.855808 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:42.855782    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" (UID: "3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:34:42.855856 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:42.855843    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:34:42.855903 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:42.855870    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:34:42.857508 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:42.857481    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" (UID: "3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:34:42.857635 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:42.857609    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-kube-api-access-msvtf" (OuterVolumeSpecName: "kube-api-access-msvtf") pod "3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" (UID: "3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6"). InnerVolumeSpecName "kube-api-access-msvtf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:34:42.956742 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:42.956707    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msvtf\" (UniqueName: \"kubernetes.io/projected/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-kube-api-access-msvtf\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:34:42.956742 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:42.956734    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:34:42.956742 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:42.956747    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:34:43.625486 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:43.625393    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms" event={"ID":"add34785-94c3-4bab-aec9-f9257d5b443e","Type":"ContainerStarted","Data":"c842ba1ce307de70adbabb25a4b4445e6be246c39af596d5e112f1be58c3155b"}
Jun 15 06:34:43.625857 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:43.625495    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms" event={"ID":"add34785-94c3-4bab-aec9-f9257d5b443e","Type":"ContainerStarted","Data":"0fa879d9d80be8593995533880dc782d7b8a22ccd0556b22451cd7d75ec885a9"}
Jun 15 06:34:43.625857 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:43.625694    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms"
Jun 15 06:34:43.627283 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:43.627260    2572 generic.go:356] "Generic (PLEG): container finished" podID="3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" containerID="ddba48d270c31b7a90cf1473c98f8bdd2d9470a15b7d23a9ec2f199485c4ce46" exitCode=0
Jun 15 06:34:43.627414 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:43.627298    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq" event={"ID":"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6","Type":"ContainerDied","Data":"ddba48d270c31b7a90cf1473c98f8bdd2d9470a15b7d23a9ec2f199485c4ce46"}
Jun 15 06:34:43.627414 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:43.627323    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq"
Jun 15 06:34:43.627414 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:43.627337    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq" event={"ID":"3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6","Type":"ContainerDied","Data":"c63115cb837924e8a1aa649c38f1488efbcc6a71a71f4d347c1fcb557fa4ef93"}
Jun 15 06:34:43.627414 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:43.627377    2572 scope.go:117] "RemoveContainer" containerID="c128171f8ffbf0546e318032dd25558bfe40bc23681465c4c72cc504dc47709f"
Jun 15 06:34:43.635144 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:43.635125    2572 scope.go:117] "RemoveContainer" containerID="ddba48d270c31b7a90cf1473c98f8bdd2d9470a15b7d23a9ec2f199485c4ce46"
Jun 15 06:34:43.643634 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:43.643618    2572 scope.go:117] "RemoveContainer" containerID="bc2bb3e9285e636eadbd009bc0ce8d928e85edaae2c8f67994bd1a651f8c10c2"
Jun 15 06:34:43.650638 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:43.650597    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms" podStartSLOduration=1.845951342 podStartE2EDuration="5.650584834s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.804633492s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:34:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:34:43.649854579 +0000 UTC m=+2798.798034578" watchObservedRunningTime="2026-06-15 06:34:43.650584834 +0000 UTC m=+2798.798764831"
Jun 15 06:34:43.651175 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:43.651159    2572 scope.go:117] "RemoveContainer" containerID="c128171f8ffbf0546e318032dd25558bfe40bc23681465c4c72cc504dc47709f"
Jun 15 06:34:43.651443 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:34:43.651421    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c128171f8ffbf0546e318032dd25558bfe40bc23681465c4c72cc504dc47709f\": container with ID starting with c128171f8ffbf0546e318032dd25558bfe40bc23681465c4c72cc504dc47709f not found: ID does not exist" containerID="c128171f8ffbf0546e318032dd25558bfe40bc23681465c4c72cc504dc47709f"
Jun 15 06:34:43.651506 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:43.651450    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c128171f8ffbf0546e318032dd25558bfe40bc23681465c4c72cc504dc47709f"} err="failed to get container status \"c128171f8ffbf0546e318032dd25558bfe40bc23681465c4c72cc504dc47709f\": rpc error: code = NotFound desc = could not find container \"c128171f8ffbf0546e318032dd25558bfe40bc23681465c4c72cc504dc47709f\": container with ID starting with c128171f8ffbf0546e318032dd25558bfe40bc23681465c4c72cc504dc47709f not found: ID does not exist"
Jun 15 06:34:43.651506 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:43.651466    2572 scope.go:117] "RemoveContainer" containerID="ddba48d270c31b7a90cf1473c98f8bdd2d9470a15b7d23a9ec2f199485c4ce46"
Jun 15 06:34:43.651700 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:34:43.651686    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddba48d270c31b7a90cf1473c98f8bdd2d9470a15b7d23a9ec2f199485c4ce46\": container with ID starting with ddba48d270c31b7a90cf1473c98f8bdd2d9470a15b7d23a9ec2f199485c4ce46 not found: ID does not exist" containerID="ddba48d270c31b7a90cf1473c98f8bdd2d9470a15b7d23a9ec2f199485c4ce46"
Jun 15 06:34:43.651745 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:43.651704    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddba48d270c31b7a90cf1473c98f8bdd2d9470a15b7d23a9ec2f199485c4ce46"} err="failed to get container status \"ddba48d270c31b7a90cf1473c98f8bdd2d9470a15b7d23a9ec2f199485c4ce46\": rpc error: code = NotFound desc = could not find container \"ddba48d270c31b7a90cf1473c98f8bdd2d9470a15b7d23a9ec2f199485c4ce46\": container with ID starting with ddba48d270c31b7a90cf1473c98f8bdd2d9470a15b7d23a9ec2f199485c4ce46 not found: ID does not exist"
Jun 15 06:34:43.651745 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:43.651716    2572 scope.go:117] "RemoveContainer" containerID="bc2bb3e9285e636eadbd009bc0ce8d928e85edaae2c8f67994bd1a651f8c10c2"
Jun 15 06:34:43.651916 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:34:43.651902    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc2bb3e9285e636eadbd009bc0ce8d928e85edaae2c8f67994bd1a651f8c10c2\": container with ID starting with bc2bb3e9285e636eadbd009bc0ce8d928e85edaae2c8f67994bd1a651f8c10c2 not found: ID does not exist" containerID="bc2bb3e9285e636eadbd009bc0ce8d928e85edaae2c8f67994bd1a651f8c10c2"
Jun 15 06:34:43.651959 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:43.651920    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc2bb3e9285e636eadbd009bc0ce8d928e85edaae2c8f67994bd1a651f8c10c2"} err="failed to get container status \"bc2bb3e9285e636eadbd009bc0ce8d928e85edaae2c8f67994bd1a651f8c10c2\": rpc error: code = NotFound desc = could not find container \"bc2bb3e9285e636eadbd009bc0ce8d928e85edaae2c8f67994bd1a651f8c10c2\": container with ID starting with bc2bb3e9285e636eadbd009bc0ce8d928e85edaae2c8f67994bd1a651f8c10c2 not found: ID does not exist"
Jun 15 06:34:43.664866 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:43.664843    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq"]
Jun 15 06:34:43.669083 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:43.669059    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-predictor-5bdbc4b78f-qh4pq"]
Jun 15 06:34:44.633940 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:44.633909    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms"
Jun 15 06:34:45.324072 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:45.324038    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" path="/var/lib/kubelet/pods/3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6/volumes"
Jun 15 06:34:50.645934 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:34:50.645904    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms"
Jun 15 06:35:20.738529 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:20.738482    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms" podUID="add34785-94c3-4bab-aec9-f9257d5b443e" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jun 15 06:35:30.649128 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:30.649097    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms"
Jun 15 06:35:38.265135 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.265091    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms"]
Jun 15 06:35:38.266017 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.265394    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms" podUID="add34785-94c3-4bab-aec9-f9257d5b443e" containerName="kserve-container" containerID="cri-o://0fa879d9d80be8593995533880dc782d7b8a22ccd0556b22451cd7d75ec885a9" gracePeriod=30
Jun 15 06:35:38.266017 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.265429    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms" podUID="add34785-94c3-4bab-aec9-f9257d5b443e" containerName="kube-rbac-proxy" containerID="cri-o://c842ba1ce307de70adbabb25a4b4445e6be246c39af596d5e112f1be58c3155b" gracePeriod=30
Jun 15 06:35:38.356820 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.356783    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"]
Jun 15 06:35:38.357148 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.357134    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" containerName="kserve-container"
Jun 15 06:35:38.357148 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.357148    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" containerName="kserve-container"
Jun 15 06:35:38.357297 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.357160    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" containerName="kube-rbac-proxy"
Jun 15 06:35:38.357297 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.357165    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" containerName="kube-rbac-proxy"
Jun 15 06:35:38.357297 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.357194    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" containerName="storage-initializer"
Jun 15 06:35:38.357297 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.357200    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" containerName="storage-initializer"
Jun 15 06:35:38.357297 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.357269    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" containerName="kube-rbac-proxy"
Jun 15 06:35:38.357297 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.357279    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e8d87fd-fa3c-41ca-bf83-f0abd41a81e6" containerName="kserve-container"
Jun 15 06:35:38.360624 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.360608    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"
Jun 15 06:35:38.362761 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.362733    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-predictor-serving-cert\""
Jun 15 06:35:38.362879 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.362734    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-runtime-kube-rbac-proxy-sar-config\""
Jun 15 06:35:38.381034 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.381008    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"]
Jun 15 06:35:38.427563 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.427526    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/663c6c55-58c8-4615-8a91-8a06c80929db-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-565f974957-7t28f\" (UID: \"663c6c55-58c8-4615-8a91-8a06c80929db\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"
Jun 15 06:35:38.427734 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.427585    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrx9q\" (UniqueName: \"kubernetes.io/projected/663c6c55-58c8-4615-8a91-8a06c80929db-kube-api-access-vrx9q\") pod \"isvc-sklearn-runtime-predictor-565f974957-7t28f\" (UID: \"663c6c55-58c8-4615-8a91-8a06c80929db\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"
Jun 15 06:35:38.427734 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.427611    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/663c6c55-58c8-4615-8a91-8a06c80929db-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-565f974957-7t28f\" (UID: \"663c6c55-58c8-4615-8a91-8a06c80929db\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"
Jun 15 06:35:38.427734 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.427631    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/663c6c55-58c8-4615-8a91-8a06c80929db-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-565f974957-7t28f\" (UID: \"663c6c55-58c8-4615-8a91-8a06c80929db\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"
Jun 15 06:35:38.427734 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.427670    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/663c6c55-58c8-4615-8a91-8a06c80929db-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-565f974957-7t28f\" (UID: \"663c6c55-58c8-4615-8a91-8a06c80929db\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"
Jun 15 06:35:38.528721 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.528633    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/663c6c55-58c8-4615-8a91-8a06c80929db-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-565f974957-7t28f\" (UID: \"663c6c55-58c8-4615-8a91-8a06c80929db\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"
Jun 15 06:35:38.528721 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.528696    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrx9q\" (UniqueName: \"kubernetes.io/projected/663c6c55-58c8-4615-8a91-8a06c80929db-kube-api-access-vrx9q\") pod \"isvc-sklearn-runtime-predictor-565f974957-7t28f\" (UID: \"663c6c55-58c8-4615-8a91-8a06c80929db\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"
Jun 15 06:35:38.528954 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.528724    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/663c6c55-58c8-4615-8a91-8a06c80929db-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-565f974957-7t28f\" (UID: \"663c6c55-58c8-4615-8a91-8a06c80929db\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"
Jun 15 06:35:38.528954 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.528754    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/663c6c55-58c8-4615-8a91-8a06c80929db-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-565f974957-7t28f\" (UID: \"663c6c55-58c8-4615-8a91-8a06c80929db\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"
Jun 15 06:35:38.528954 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.528800    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/663c6c55-58c8-4615-8a91-8a06c80929db-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-565f974957-7t28f\" (UID: \"663c6c55-58c8-4615-8a91-8a06c80929db\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"
Jun 15 06:35:38.529114 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:35:38.528956    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-serving-cert: secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Jun 15 06:35:38.529114 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:35:38.529045    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/663c6c55-58c8-4615-8a91-8a06c80929db-proxy-tls podName:663c6c55-58c8-4615-8a91-8a06c80929db nodeName:}" failed. No retries permitted until 2026-06-15 06:35:39.029022165 +0000 UTC m=+2854.177202162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/663c6c55-58c8-4615-8a91-8a06c80929db-proxy-tls") pod "isvc-sklearn-runtime-predictor-565f974957-7t28f" (UID: "663c6c55-58c8-4615-8a91-8a06c80929db") : secret "isvc-sklearn-runtime-predictor-serving-cert" not found
Jun 15 06:35:38.529114 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.529065    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/663c6c55-58c8-4615-8a91-8a06c80929db-kserve-provision-location\") pod \"isvc-sklearn-runtime-predictor-565f974957-7t28f\" (UID: \"663c6c55-58c8-4615-8a91-8a06c80929db\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"
Jun 15 06:35:38.529383 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.529348    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/663c6c55-58c8-4615-8a91-8a06c80929db-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-runtime-predictor-565f974957-7t28f\" (UID: \"663c6c55-58c8-4615-8a91-8a06c80929db\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"
Jun 15 06:35:38.529461 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.529446    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/663c6c55-58c8-4615-8a91-8a06c80929db-cabundle-cert\") pod \"isvc-sklearn-runtime-predictor-565f974957-7t28f\" (UID: \"663c6c55-58c8-4615-8a91-8a06c80929db\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"
Jun 15 06:35:38.540342 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.540322    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrx9q\" (UniqueName: \"kubernetes.io/projected/663c6c55-58c8-4615-8a91-8a06c80929db-kube-api-access-vrx9q\") pod \"isvc-sklearn-runtime-predictor-565f974957-7t28f\" (UID: \"663c6c55-58c8-4615-8a91-8a06c80929db\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"
Jun 15 06:35:38.896499 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.896469    2572 generic.go:356] "Generic (PLEG): container finished" podID="add34785-94c3-4bab-aec9-f9257d5b443e" containerID="c842ba1ce307de70adbabb25a4b4445e6be246c39af596d5e112f1be58c3155b" exitCode=2
Jun 15 06:35:38.896669 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:38.896539    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms" event={"ID":"add34785-94c3-4bab-aec9-f9257d5b443e","Type":"ContainerDied","Data":"c842ba1ce307de70adbabb25a4b4445e6be246c39af596d5e112f1be58c3155b"}
Jun 15 06:35:39.032602 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:39.032564    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/663c6c55-58c8-4615-8a91-8a06c80929db-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-565f974957-7t28f\" (UID: \"663c6c55-58c8-4615-8a91-8a06c80929db\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"
Jun 15 06:35:39.034922 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:39.034895    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/663c6c55-58c8-4615-8a91-8a06c80929db-proxy-tls\") pod \"isvc-sklearn-runtime-predictor-565f974957-7t28f\" (UID: \"663c6c55-58c8-4615-8a91-8a06c80929db\") " pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"
Jun 15 06:35:39.270665 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:39.270573    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"
Jun 15 06:35:39.403206 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:39.403183    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"]
Jun 15 06:35:39.405636 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:35:39.405608    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663c6c55_58c8_4615_8a91_8a06c80929db.slice/crio-a2fec5fb01cda9c8be2e05ec5a12d67106b7bf004571de339a008b52b2a237ff WatchSource:0}: Error finding container a2fec5fb01cda9c8be2e05ec5a12d67106b7bf004571de339a008b52b2a237ff: Status 404 returned error can't find the container with id a2fec5fb01cda9c8be2e05ec5a12d67106b7bf004571de339a008b52b2a237ff
Jun 15 06:35:39.903439 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:39.903402    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f" event={"ID":"663c6c55-58c8-4615-8a91-8a06c80929db","Type":"ContainerStarted","Data":"870543d8b512eb332bde849d41eddff8baa42f3528e4c8743b52d7a5daf15649"}
Jun 15 06:35:39.903621 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:39.903475    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f" event={"ID":"663c6c55-58c8-4615-8a91-8a06c80929db","Type":"ContainerStarted","Data":"a2fec5fb01cda9c8be2e05ec5a12d67106b7bf004571de339a008b52b2a237ff"}
Jun 15 06:35:40.641909 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:40.641862    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms" podUID="add34785-94c3-4bab-aec9-f9257d5b443e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Jun 15 06:35:41.687576 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:41.687529    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms" podUID="add34785-94c3-4bab-aec9-f9257d5b443e" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.51:8080/v2/models/sklearn-v2-mlserver/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jun 15 06:35:45.710696 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.710673    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms"
Jun 15 06:35:45.787669 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.787583    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/add34785-94c3-4bab-aec9-f9257d5b443e-kube-rbac-proxy-sar-config\") pod \"add34785-94c3-4bab-aec9-f9257d5b443e\" (UID: \"add34785-94c3-4bab-aec9-f9257d5b443e\") "
Jun 15 06:35:45.787669 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.787649    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/add34785-94c3-4bab-aec9-f9257d5b443e-cabundle-cert\") pod \"add34785-94c3-4bab-aec9-f9257d5b443e\" (UID: \"add34785-94c3-4bab-aec9-f9257d5b443e\") "
Jun 15 06:35:45.787884 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.787674    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/add34785-94c3-4bab-aec9-f9257d5b443e-kserve-provision-location\") pod \"add34785-94c3-4bab-aec9-f9257d5b443e\" (UID: \"add34785-94c3-4bab-aec9-f9257d5b443e\") "
Jun 15 06:35:45.787884 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.787758    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqm2b\" (UniqueName: \"kubernetes.io/projected/add34785-94c3-4bab-aec9-f9257d5b443e-kube-api-access-zqm2b\") pod \"add34785-94c3-4bab-aec9-f9257d5b443e\" (UID: \"add34785-94c3-4bab-aec9-f9257d5b443e\") "
Jun 15 06:35:45.787884 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.787813    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/add34785-94c3-4bab-aec9-f9257d5b443e-proxy-tls\") pod \"add34785-94c3-4bab-aec9-f9257d5b443e\" (UID: \"add34785-94c3-4bab-aec9-f9257d5b443e\") "
Jun 15 06:35:45.788063 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.788035    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add34785-94c3-4bab-aec9-f9257d5b443e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "add34785-94c3-4bab-aec9-f9257d5b443e" (UID: "add34785-94c3-4bab-aec9-f9257d5b443e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:35:45.788153 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.788095    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add34785-94c3-4bab-aec9-f9257d5b443e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "add34785-94c3-4bab-aec9-f9257d5b443e" (UID: "add34785-94c3-4bab-aec9-f9257d5b443e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:35:45.788153 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.788116    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/add34785-94c3-4bab-aec9-f9257d5b443e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "add34785-94c3-4bab-aec9-f9257d5b443e" (UID: "add34785-94c3-4bab-aec9-f9257d5b443e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:35:45.789883 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.789863    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/add34785-94c3-4bab-aec9-f9257d5b443e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "add34785-94c3-4bab-aec9-f9257d5b443e" (UID: "add34785-94c3-4bab-aec9-f9257d5b443e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:35:45.790065 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.790036    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/add34785-94c3-4bab-aec9-f9257d5b443e-kube-api-access-zqm2b" (OuterVolumeSpecName: "kube-api-access-zqm2b") pod "add34785-94c3-4bab-aec9-f9257d5b443e" (UID: "add34785-94c3-4bab-aec9-f9257d5b443e"). InnerVolumeSpecName "kube-api-access-zqm2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:35:45.888906 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.888868    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqm2b\" (UniqueName: \"kubernetes.io/projected/add34785-94c3-4bab-aec9-f9257d5b443e-kube-api-access-zqm2b\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:35:45.888906 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.888904    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/add34785-94c3-4bab-aec9-f9257d5b443e-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:35:45.889147 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.888920    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/add34785-94c3-4bab-aec9-f9257d5b443e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:35:45.889147 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.888932    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/add34785-94c3-4bab-aec9-f9257d5b443e-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:35:45.889147 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.888944    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/add34785-94c3-4bab-aec9-f9257d5b443e-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:35:45.934023 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.933992    2572 generic.go:356] "Generic (PLEG): container finished" podID="add34785-94c3-4bab-aec9-f9257d5b443e" containerID="0fa879d9d80be8593995533880dc782d7b8a22ccd0556b22451cd7d75ec885a9" exitCode=0
Jun 15 06:35:45.934186 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.934049    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms"
Jun 15 06:35:45.934186 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.934076    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms" event={"ID":"add34785-94c3-4bab-aec9-f9257d5b443e","Type":"ContainerDied","Data":"0fa879d9d80be8593995533880dc782d7b8a22ccd0556b22451cd7d75ec885a9"}
Jun 15 06:35:45.934186 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.934150    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms" event={"ID":"add34785-94c3-4bab-aec9-f9257d5b443e","Type":"ContainerDied","Data":"dde7c2493b4cece61a23a668eb6a040b176937560a5b6c95c2762f0c17703dd2"}
Jun 15 06:35:45.934323 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.934186    2572 scope.go:117] "RemoveContainer" containerID="c842ba1ce307de70adbabb25a4b4445e6be246c39af596d5e112f1be58c3155b"
Jun 15 06:35:45.935743 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.935721    2572 generic.go:356] "Generic (PLEG): container finished" podID="663c6c55-58c8-4615-8a91-8a06c80929db" containerID="870543d8b512eb332bde849d41eddff8baa42f3528e4c8743b52d7a5daf15649" exitCode=0
Jun 15 06:35:45.935860 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.935760    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f" event={"ID":"663c6c55-58c8-4615-8a91-8a06c80929db","Type":"ContainerDied","Data":"870543d8b512eb332bde849d41eddff8baa42f3528e4c8743b52d7a5daf15649"}
Jun 15 06:35:45.943671 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.943650    2572 scope.go:117] "RemoveContainer" containerID="0fa879d9d80be8593995533880dc782d7b8a22ccd0556b22451cd7d75ec885a9"
Jun 15 06:35:45.951986 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.951968    2572 scope.go:117] "RemoveContainer" containerID="b3504b509e91366f24abc0471e0c9ededc804d5ed487be731e5a97bb4d291117"
Jun 15 06:35:45.960731 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.960712    2572 scope.go:117] "RemoveContainer" containerID="c842ba1ce307de70adbabb25a4b4445e6be246c39af596d5e112f1be58c3155b"
Jun 15 06:35:45.960983 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:35:45.960960    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c842ba1ce307de70adbabb25a4b4445e6be246c39af596d5e112f1be58c3155b\": container with ID starting with c842ba1ce307de70adbabb25a4b4445e6be246c39af596d5e112f1be58c3155b not found: ID does not exist" containerID="c842ba1ce307de70adbabb25a4b4445e6be246c39af596d5e112f1be58c3155b"
Jun 15 06:35:45.961086 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.960991    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c842ba1ce307de70adbabb25a4b4445e6be246c39af596d5e112f1be58c3155b"} err="failed to get container status \"c842ba1ce307de70adbabb25a4b4445e6be246c39af596d5e112f1be58c3155b\": rpc error: code = NotFound desc = could not find container \"c842ba1ce307de70adbabb25a4b4445e6be246c39af596d5e112f1be58c3155b\": container with ID starting with c842ba1ce307de70adbabb25a4b4445e6be246c39af596d5e112f1be58c3155b not found: ID does not exist"
Jun 15 06:35:45.961086 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.961008    2572 scope.go:117] "RemoveContainer" containerID="0fa879d9d80be8593995533880dc782d7b8a22ccd0556b22451cd7d75ec885a9"
Jun 15 06:35:45.961251 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:35:45.961236    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fa879d9d80be8593995533880dc782d7b8a22ccd0556b22451cd7d75ec885a9\": container with ID starting with 0fa879d9d80be8593995533880dc782d7b8a22ccd0556b22451cd7d75ec885a9 not found: ID does not exist" containerID="0fa879d9d80be8593995533880dc782d7b8a22ccd0556b22451cd7d75ec885a9"
Jun 15 06:35:45.961294 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.961254    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fa879d9d80be8593995533880dc782d7b8a22ccd0556b22451cd7d75ec885a9"} err="failed to get container status \"0fa879d9d80be8593995533880dc782d7b8a22ccd0556b22451cd7d75ec885a9\": rpc error: code = NotFound desc = could not find container \"0fa879d9d80be8593995533880dc782d7b8a22ccd0556b22451cd7d75ec885a9\": container with ID starting with 0fa879d9d80be8593995533880dc782d7b8a22ccd0556b22451cd7d75ec885a9 not found: ID does not exist"
Jun 15 06:35:45.961294 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.961266    2572 scope.go:117] "RemoveContainer" containerID="b3504b509e91366f24abc0471e0c9ededc804d5ed487be731e5a97bb4d291117"
Jun 15 06:35:45.961530 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:35:45.961506    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3504b509e91366f24abc0471e0c9ededc804d5ed487be731e5a97bb4d291117\": container with ID starting with b3504b509e91366f24abc0471e0c9ededc804d5ed487be731e5a97bb4d291117 not found: ID does not exist" containerID="b3504b509e91366f24abc0471e0c9ededc804d5ed487be731e5a97bb4d291117"
Jun 15 06:35:45.961579 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.961536    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3504b509e91366f24abc0471e0c9ededc804d5ed487be731e5a97bb4d291117"} err="failed to get container status \"b3504b509e91366f24abc0471e0c9ededc804d5ed487be731e5a97bb4d291117\": rpc error: code = NotFound desc = could not find container \"b3504b509e91366f24abc0471e0c9ededc804d5ed487be731e5a97bb4d291117\": container with ID starting with b3504b509e91366f24abc0471e0c9ededc804d5ed487be731e5a97bb4d291117 not found: ID does not exist"
Jun 15 06:35:45.978430 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.978404    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms"]
Jun 15 06:35:45.983266 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:45.983242    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms"]
Jun 15 06:35:46.641637 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:46.641598    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sklearn-v2-mlserver-predictor-7f568cdc6b-cl7ms" podUID="add34785-94c3-4bab-aec9-f9257d5b443e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": context deadline exceeded"
Jun 15 06:35:46.944437 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:46.944337    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f" event={"ID":"663c6c55-58c8-4615-8a91-8a06c80929db","Type":"ContainerStarted","Data":"ff244939d67e0c834821d87678a2d466b64129af9bbf111866dc0a1ddf2e7ae3"}
Jun 15 06:35:46.944437 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:46.944418    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f" event={"ID":"663c6c55-58c8-4615-8a91-8a06c80929db","Type":"ContainerStarted","Data":"6895c125732187971b31ee1a2b85bdb93b19168c3357bc1cb97c1b302ab557fa"}
Jun 15 06:35:46.944975 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:46.944601    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"
Jun 15 06:35:46.965589 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:46.965537    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f" podStartSLOduration=3.286147587 podStartE2EDuration="8.965522761s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.679375174s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:35:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:35:46.963565657 +0000 UTC m=+2862.111745655" watchObservedRunningTime="2026-06-15 06:35:46.965522761 +0000 UTC m=+2862.113702757"
Jun 15 06:35:47.324382 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:47.324269    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="add34785-94c3-4bab-aec9-f9257d5b443e" path="/var/lib/kubelet/pods/add34785-94c3-4bab-aec9-f9257d5b443e/volumes"
Jun 15 06:35:47.950309 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:47.950283    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"
Jun 15 06:35:47.951491 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:47.951456    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f" podUID="663c6c55-58c8-4615-8a91-8a06c80929db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jun 15 06:35:48.955571 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:48.955529    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f" podUID="663c6c55-58c8-4615-8a91-8a06c80929db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jun 15 06:35:53.961450 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:53.961406    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"
Jun 15 06:35:53.961986 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:35:53.961958    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f" podUID="663c6c55-58c8-4615-8a91-8a06c80929db" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jun 15 06:36:03.962800 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:03.962765    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"
Jun 15 06:36:15.250140 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.250074    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-runtime-predictor-565f974957-7t28f_663c6c55-58c8-4615-8a91-8a06c80929db/kserve-container/0.log"
Jun 15 06:36:15.447130 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.444438    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"]
Jun 15 06:36:15.447130 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.444818    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f" podUID="663c6c55-58c8-4615-8a91-8a06c80929db" containerName="kserve-container" containerID="cri-o://6895c125732187971b31ee1a2b85bdb93b19168c3357bc1cb97c1b302ab557fa" gracePeriod=30
Jun 15 06:36:15.447130 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.445086    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f" podUID="663c6c55-58c8-4615-8a91-8a06c80929db" containerName="kube-rbac-proxy" containerID="cri-o://ff244939d67e0c834821d87678a2d466b64129af9bbf111866dc0a1ddf2e7ae3" gracePeriod=30
Jun 15 06:36:15.465342 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.465302    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"]
Jun 15 06:36:15.465733 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.465718    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="add34785-94c3-4bab-aec9-f9257d5b443e" containerName="storage-initializer"
Jun 15 06:36:15.465779 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.465735    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="add34785-94c3-4bab-aec9-f9257d5b443e" containerName="storage-initializer"
Jun 15 06:36:15.465779 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.465750    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="add34785-94c3-4bab-aec9-f9257d5b443e" containerName="kserve-container"
Jun 15 06:36:15.465779 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.465755    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="add34785-94c3-4bab-aec9-f9257d5b443e" containerName="kserve-container"
Jun 15 06:36:15.465779 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.465769    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="add34785-94c3-4bab-aec9-f9257d5b443e" containerName="kube-rbac-proxy"
Jun 15 06:36:15.465779 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.465778    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="add34785-94c3-4bab-aec9-f9257d5b443e" containerName="kube-rbac-proxy"
Jun 15 06:36:15.465978 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.465837    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="add34785-94c3-4bab-aec9-f9257d5b443e" containerName="kserve-container"
Jun 15 06:36:15.465978 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.465847    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="add34785-94c3-4bab-aec9-f9257d5b443e" containerName="kube-rbac-proxy"
Jun 15 06:36:15.469179 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.469159    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"
Jun 15 06:36:15.471105 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.471083    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-kube-rbac-proxy-sar-config\""
Jun 15 06:36:15.471205 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.471101    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-runtime-predictor-serving-cert\""
Jun 15 06:36:15.479096 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.479075    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"]
Jun 15 06:36:15.557384 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.557290    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65lcv\" (UniqueName: \"kubernetes.io/projected/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-kube-api-access-65lcv\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w\" (UID: \"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"
Jun 15 06:36:15.557533 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.557401    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w\" (UID: \"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"
Jun 15 06:36:15.557533 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.557443    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w\" (UID: \"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"
Jun 15 06:36:15.557533 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.557468    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w\" (UID: \"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"
Jun 15 06:36:15.557533 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.557488    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w\" (UID: \"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"
Jun 15 06:36:15.658703 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.658670    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w\" (UID: \"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"
Jun 15 06:36:15.658703 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.658708    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w\" (UID: \"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"
Jun 15 06:36:15.658939 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.658727    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w\" (UID: \"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"
Jun 15 06:36:15.658939 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.658796    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65lcv\" (UniqueName: \"kubernetes.io/projected/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-kube-api-access-65lcv\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w\" (UID: \"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"
Jun 15 06:36:15.658939 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.658852    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w\" (UID: \"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"
Jun 15 06:36:15.658939 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:36:15.658914    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-serving-cert: secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Jun 15 06:36:15.659123 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:36:15.658994    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-proxy-tls podName:4b5e9f23-8aa8-45e7-b7b5-36c72e99c800 nodeName:}" failed. No retries permitted until 2026-06-15 06:36:16.158973141 +0000 UTC m=+2891.307153119 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-proxy-tls") pod "isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w" (UID: "4b5e9f23-8aa8-45e7-b7b5-36c72e99c800") : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Jun 15 06:36:15.659214 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.659197    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-kserve-provision-location\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w\" (UID: \"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"
Jun 15 06:36:15.659507 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.659489    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w\" (UID: \"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"
Jun 15 06:36:15.659565 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.659542    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-cabundle-cert\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w\" (UID: \"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"
Jun 15 06:36:15.667838 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:15.667816    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65lcv\" (UniqueName: \"kubernetes.io/projected/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-kube-api-access-65lcv\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w\" (UID: \"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"
Jun 15 06:36:16.087039 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:16.086997    2572 generic.go:356] "Generic (PLEG): container finished" podID="663c6c55-58c8-4615-8a91-8a06c80929db" containerID="ff244939d67e0c834821d87678a2d466b64129af9bbf111866dc0a1ddf2e7ae3" exitCode=2
Jun 15 06:36:16.087216 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:16.087069    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f" event={"ID":"663c6c55-58c8-4615-8a91-8a06c80929db","Type":"ContainerDied","Data":"ff244939d67e0c834821d87678a2d466b64129af9bbf111866dc0a1ddf2e7ae3"}
Jun 15 06:36:16.163101 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:16.163069    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w\" (UID: \"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"
Jun 15 06:36:16.163302 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:36:16.163245    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-serving-cert: secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Jun 15 06:36:16.163394 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:36:16.163318    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-proxy-tls podName:4b5e9f23-8aa8-45e7-b7b5-36c72e99c800 nodeName:}" failed. No retries permitted until 2026-06-15 06:36:17.16329742 +0000 UTC m=+2892.311477397 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-proxy-tls") pod "isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w" (UID: "4b5e9f23-8aa8-45e7-b7b5-36c72e99c800") : secret "isvc-sklearn-v2-runtime-predictor-serving-cert" not found
Jun 15 06:36:16.488587 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:16.488561    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"
Jun 15 06:36:16.566532 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:16.566496    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/663c6c55-58c8-4615-8a91-8a06c80929db-kserve-provision-location\") pod \"663c6c55-58c8-4615-8a91-8a06c80929db\" (UID: \"663c6c55-58c8-4615-8a91-8a06c80929db\") "
Jun 15 06:36:16.566727 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:16.566545    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/663c6c55-58c8-4615-8a91-8a06c80929db-kube-rbac-proxy-sar-config\") pod \"663c6c55-58c8-4615-8a91-8a06c80929db\" (UID: \"663c6c55-58c8-4615-8a91-8a06c80929db\") "
Jun 15 06:36:16.566727 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:16.566596    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/663c6c55-58c8-4615-8a91-8a06c80929db-proxy-tls\") pod \"663c6c55-58c8-4615-8a91-8a06c80929db\" (UID: \"663c6c55-58c8-4615-8a91-8a06c80929db\") "
Jun 15 06:36:16.566727 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:16.566616    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/663c6c55-58c8-4615-8a91-8a06c80929db-cabundle-cert\") pod \"663c6c55-58c8-4615-8a91-8a06c80929db\" (UID: \"663c6c55-58c8-4615-8a91-8a06c80929db\") "
Jun 15 06:36:16.566727 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:16.566679    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrx9q\" (UniqueName: \"kubernetes.io/projected/663c6c55-58c8-4615-8a91-8a06c80929db-kube-api-access-vrx9q\") pod \"663c6c55-58c8-4615-8a91-8a06c80929db\" (UID: \"663c6c55-58c8-4615-8a91-8a06c80929db\") "
Jun 15 06:36:16.566999 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:16.566968    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/663c6c55-58c8-4615-8a91-8a06c80929db-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "663c6c55-58c8-4615-8a91-8a06c80929db" (UID: "663c6c55-58c8-4615-8a91-8a06c80929db"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:36:16.567061 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:16.566997    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/663c6c55-58c8-4615-8a91-8a06c80929db-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "663c6c55-58c8-4615-8a91-8a06c80929db" (UID: "663c6c55-58c8-4615-8a91-8a06c80929db"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:36:16.568743 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:16.568716    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/663c6c55-58c8-4615-8a91-8a06c80929db-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "663c6c55-58c8-4615-8a91-8a06c80929db" (UID: "663c6c55-58c8-4615-8a91-8a06c80929db"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:36:16.568860 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:16.568811    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/663c6c55-58c8-4615-8a91-8a06c80929db-kube-api-access-vrx9q" (OuterVolumeSpecName: "kube-api-access-vrx9q") pod "663c6c55-58c8-4615-8a91-8a06c80929db" (UID: "663c6c55-58c8-4615-8a91-8a06c80929db"). InnerVolumeSpecName "kube-api-access-vrx9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:36:16.594701 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:16.594675    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/663c6c55-58c8-4615-8a91-8a06c80929db-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "663c6c55-58c8-4615-8a91-8a06c80929db" (UID: "663c6c55-58c8-4615-8a91-8a06c80929db"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:36:16.667842 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:16.667810    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/663c6c55-58c8-4615-8a91-8a06c80929db-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:36:16.667842 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:16.667837    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/663c6c55-58c8-4615-8a91-8a06c80929db-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:36:16.668005 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:16.667850    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/663c6c55-58c8-4615-8a91-8a06c80929db-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:36:16.668005 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:16.667862    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrx9q\" (UniqueName: \"kubernetes.io/projected/663c6c55-58c8-4615-8a91-8a06c80929db-kube-api-access-vrx9q\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:36:16.668005 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:16.667873    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/663c6c55-58c8-4615-8a91-8a06c80929db-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:36:17.093722 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:17.093637    2572 generic.go:356] "Generic (PLEG): container finished" podID="663c6c55-58c8-4615-8a91-8a06c80929db" containerID="6895c125732187971b31ee1a2b85bdb93b19168c3357bc1cb97c1b302ab557fa" exitCode=0
Jun 15 06:36:17.093722 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:17.093691    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"
Jun 15 06:36:17.093903 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:17.093725    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f" event={"ID":"663c6c55-58c8-4615-8a91-8a06c80929db","Type":"ContainerDied","Data":"6895c125732187971b31ee1a2b85bdb93b19168c3357bc1cb97c1b302ab557fa"}
Jun 15 06:36:17.093903 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:17.093799    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f" event={"ID":"663c6c55-58c8-4615-8a91-8a06c80929db","Type":"ContainerDied","Data":"a2fec5fb01cda9c8be2e05ec5a12d67106b7bf004571de339a008b52b2a237ff"}
Jun 15 06:36:17.093903 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:17.093839    2572 scope.go:117] "RemoveContainer" containerID="ff244939d67e0c834821d87678a2d466b64129af9bbf111866dc0a1ddf2e7ae3"
Jun 15 06:36:17.103657 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:17.103638    2572 scope.go:117] "RemoveContainer" containerID="6895c125732187971b31ee1a2b85bdb93b19168c3357bc1cb97c1b302ab557fa"
Jun 15 06:36:17.111390 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:17.111352    2572 scope.go:117] "RemoveContainer" containerID="870543d8b512eb332bde849d41eddff8baa42f3528e4c8743b52d7a5daf15649"
Jun 15 06:36:17.121443 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:17.121423    2572 scope.go:117] "RemoveContainer" containerID="ff244939d67e0c834821d87678a2d466b64129af9bbf111866dc0a1ddf2e7ae3"
Jun 15 06:36:17.121721 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:17.121699    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"]
Jun 15 06:36:17.121801 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:36:17.121722    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff244939d67e0c834821d87678a2d466b64129af9bbf111866dc0a1ddf2e7ae3\": container with ID starting with ff244939d67e0c834821d87678a2d466b64129af9bbf111866dc0a1ddf2e7ae3 not found: ID does not exist" containerID="ff244939d67e0c834821d87678a2d466b64129af9bbf111866dc0a1ddf2e7ae3"
Jun 15 06:36:17.121801 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:17.121754    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff244939d67e0c834821d87678a2d466b64129af9bbf111866dc0a1ddf2e7ae3"} err="failed to get container status \"ff244939d67e0c834821d87678a2d466b64129af9bbf111866dc0a1ddf2e7ae3\": rpc error: code = NotFound desc = could not find container \"ff244939d67e0c834821d87678a2d466b64129af9bbf111866dc0a1ddf2e7ae3\": container with ID starting with ff244939d67e0c834821d87678a2d466b64129af9bbf111866dc0a1ddf2e7ae3 not found: ID does not exist"
Jun 15 06:36:17.121801 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:17.121782    2572 scope.go:117] "RemoveContainer" containerID="6895c125732187971b31ee1a2b85bdb93b19168c3357bc1cb97c1b302ab557fa"
Jun 15 06:36:17.122035 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:36:17.122019    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6895c125732187971b31ee1a2b85bdb93b19168c3357bc1cb97c1b302ab557fa\": container with ID starting with 6895c125732187971b31ee1a2b85bdb93b19168c3357bc1cb97c1b302ab557fa not found: ID does not exist" containerID="6895c125732187971b31ee1a2b85bdb93b19168c3357bc1cb97c1b302ab557fa"
Jun 15 06:36:17.122072 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:17.122044    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6895c125732187971b31ee1a2b85bdb93b19168c3357bc1cb97c1b302ab557fa"} err="failed to get container status \"6895c125732187971b31ee1a2b85bdb93b19168c3357bc1cb97c1b302ab557fa\": rpc error: code = NotFound desc = could not find container \"6895c125732187971b31ee1a2b85bdb93b19168c3357bc1cb97c1b302ab557fa\": container with ID starting with 6895c125732187971b31ee1a2b85bdb93b19168c3357bc1cb97c1b302ab557fa not found: ID does not exist"
Jun 15 06:36:17.122072 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:17.122061    2572 scope.go:117] "RemoveContainer" containerID="870543d8b512eb332bde849d41eddff8baa42f3528e4c8743b52d7a5daf15649"
Jun 15 06:36:17.122269 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:36:17.122252    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"870543d8b512eb332bde849d41eddff8baa42f3528e4c8743b52d7a5daf15649\": container with ID starting with 870543d8b512eb332bde849d41eddff8baa42f3528e4c8743b52d7a5daf15649 not found: ID does not exist" containerID="870543d8b512eb332bde849d41eddff8baa42f3528e4c8743b52d7a5daf15649"
Jun 15 06:36:17.122332 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:17.122277    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"870543d8b512eb332bde849d41eddff8baa42f3528e4c8743b52d7a5daf15649"} err="failed to get container status \"870543d8b512eb332bde849d41eddff8baa42f3528e4c8743b52d7a5daf15649\": rpc error: code = NotFound desc = could not find container \"870543d8b512eb332bde849d41eddff8baa42f3528e4c8743b52d7a5daf15649\": container with ID starting with 870543d8b512eb332bde849d41eddff8baa42f3528e4c8743b52d7a5daf15649 not found: ID does not exist"
Jun 15 06:36:17.126611 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:17.126591    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-runtime-predictor-565f974957-7t28f"]
Jun 15 06:36:17.171740 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:17.171708    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w\" (UID: \"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"
Jun 15 06:36:17.174111 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:17.174087    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-proxy-tls\") pod \"isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w\" (UID: \"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"
Jun 15 06:36:17.279606 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:17.279565    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"
Jun 15 06:36:17.325300 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:17.325266    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="663c6c55-58c8-4615-8a91-8a06c80929db" path="/var/lib/kubelet/pods/663c6c55-58c8-4615-8a91-8a06c80929db/volumes"
Jun 15 06:36:17.417816 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:17.417788    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"]
Jun 15 06:36:17.419953 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:36:17.419930    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b5e9f23_8aa8_45e7_b7b5_36c72e99c800.slice/crio-5b584d8b37440377f357463e99ff2aeb8e03c5bdbc1da5995ebd828c1a61b381 WatchSource:0}: Error finding container 5b584d8b37440377f357463e99ff2aeb8e03c5bdbc1da5995ebd828c1a61b381: Status 404 returned error can't find the container with id 5b584d8b37440377f357463e99ff2aeb8e03c5bdbc1da5995ebd828c1a61b381
Jun 15 06:36:18.100301 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:18.100264    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w" event={"ID":"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800","Type":"ContainerStarted","Data":"804963d63b1837a99b7f28f87568bf14925fa3c79d79296c1782d74a78f3e805"}
Jun 15 06:36:18.100721 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:18.100321    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w" event={"ID":"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800","Type":"ContainerStarted","Data":"5b584d8b37440377f357463e99ff2aeb8e03c5bdbc1da5995ebd828c1a61b381"}
Jun 15 06:36:22.119745 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:22.119710    2572 generic.go:356] "Generic (PLEG): container finished" podID="4b5e9f23-8aa8-45e7-b7b5-36c72e99c800" containerID="804963d63b1837a99b7f28f87568bf14925fa3c79d79296c1782d74a78f3e805" exitCode=0
Jun 15 06:36:22.120120 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:22.119788    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w" event={"ID":"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800","Type":"ContainerDied","Data":"804963d63b1837a99b7f28f87568bf14925fa3c79d79296c1782d74a78f3e805"}
Jun 15 06:36:23.126535 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:23.126500    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w" event={"ID":"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800","Type":"ContainerStarted","Data":"1f7ec8643c1562d3ee625a9e1d96cc13048e2f35506f7f3e6d777cfefcb82ea5"}
Jun 15 06:36:23.126910 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:23.126560    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w" event={"ID":"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800","Type":"ContainerStarted","Data":"32b012570a6c6984ab02abe9c1899366963748eea9021080ea5336c72dcb5963"}
Jun 15 06:36:23.126910 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:23.126741    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"
Jun 15 06:36:23.149012 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:23.148965    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w" podStartSLOduration=4.485015621 podStartE2EDuration="8.148950272s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.663934651s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:36:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:36:23.148074039 +0000 UTC m=+2898.296254050" watchObservedRunningTime="2026-06-15 06:36:23.148950272 +0000 UTC m=+2898.297130269"
Jun 15 06:36:24.131543 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:24.131511    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"
Jun 15 06:36:30.141656 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:36:30.141626    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"
Jun 15 06:37:00.238214 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:00.238158    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w" podUID="4b5e9f23-8aa8-45e7-b7b5-36c72e99c800" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jun 15 06:37:10.144898 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:10.144870    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"
Jun 15 06:37:15.564226 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.564069    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"]
Jun 15 06:37:15.564684 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.564569    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w" podUID="4b5e9f23-8aa8-45e7-b7b5-36c72e99c800" containerName="kserve-container" containerID="cri-o://32b012570a6c6984ab02abe9c1899366963748eea9021080ea5336c72dcb5963" gracePeriod=30
Jun 15 06:37:15.564684 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.564604    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w" podUID="4b5e9f23-8aa8-45e7-b7b5-36c72e99c800" containerName="kube-rbac-proxy" containerID="cri-o://1f7ec8643c1562d3ee625a9e1d96cc13048e2f35506f7f3e6d777cfefcb82ea5" gracePeriod=30
Jun 15 06:37:15.640032 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.639980    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n"]
Jun 15 06:37:15.643933 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.641128    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="663c6c55-58c8-4615-8a91-8a06c80929db" containerName="kube-rbac-proxy"
Jun 15 06:37:15.643933 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.641158    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="663c6c55-58c8-4615-8a91-8a06c80929db" containerName="kube-rbac-proxy"
Jun 15 06:37:15.643933 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.641189    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="663c6c55-58c8-4615-8a91-8a06c80929db" containerName="storage-initializer"
Jun 15 06:37:15.643933 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.641199    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="663c6c55-58c8-4615-8a91-8a06c80929db" containerName="storage-initializer"
Jun 15 06:37:15.643933 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.641220    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="663c6c55-58c8-4615-8a91-8a06c80929db" containerName="kserve-container"
Jun 15 06:37:15.643933 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.641230    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="663c6c55-58c8-4615-8a91-8a06c80929db" containerName="kserve-container"
Jun 15 06:37:15.643933 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.641474    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="663c6c55-58c8-4615-8a91-8a06c80929db" containerName="kserve-container"
Jun 15 06:37:15.643933 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.641498    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="663c6c55-58c8-4615-8a91-8a06c80929db" containerName="kube-rbac-proxy"
Jun 15 06:37:15.651583 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.651514    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n"
Jun 15 06:37:15.653950 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.653888    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Jun 15 06:37:15.653950 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.653907    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Jun 15 06:37:15.655336 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.655297    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n"]
Jun 15 06:37:15.789607 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.789576    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1839a90-d614-43ed-883d-c5f37a1cebf5-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-b695d4db9-f2t2n\" (UID: \"d1839a90-d614-43ed-883d-c5f37a1cebf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n"
Jun 15 06:37:15.789829 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.789631    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1839a90-d614-43ed-883d-c5f37a1cebf5-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-b695d4db9-f2t2n\" (UID: \"d1839a90-d614-43ed-883d-c5f37a1cebf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n"
Jun 15 06:37:15.789829 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.789714    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1839a90-d614-43ed-883d-c5f37a1cebf5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-b695d4db9-f2t2n\" (UID: \"d1839a90-d614-43ed-883d-c5f37a1cebf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n"
Jun 15 06:37:15.789829 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.789786    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1839a90-d614-43ed-883d-c5f37a1cebf5-proxy-tls\") pod \"isvc-sklearn-v2-predictor-b695d4db9-f2t2n\" (UID: \"d1839a90-d614-43ed-883d-c5f37a1cebf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n"
Jun 15 06:37:15.789829 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.789815    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6c28\" (UniqueName: \"kubernetes.io/projected/d1839a90-d614-43ed-883d-c5f37a1cebf5-kube-api-access-c6c28\") pod \"isvc-sklearn-v2-predictor-b695d4db9-f2t2n\" (UID: \"d1839a90-d614-43ed-883d-c5f37a1cebf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n"
Jun 15 06:37:15.890510 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.890485    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1839a90-d614-43ed-883d-c5f37a1cebf5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-b695d4db9-f2t2n\" (UID: \"d1839a90-d614-43ed-883d-c5f37a1cebf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n"
Jun 15 06:37:15.890663 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.890535    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1839a90-d614-43ed-883d-c5f37a1cebf5-proxy-tls\") pod \"isvc-sklearn-v2-predictor-b695d4db9-f2t2n\" (UID: \"d1839a90-d614-43ed-883d-c5f37a1cebf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n"
Jun 15 06:37:15.890710 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.890649    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6c28\" (UniqueName: \"kubernetes.io/projected/d1839a90-d614-43ed-883d-c5f37a1cebf5-kube-api-access-c6c28\") pod \"isvc-sklearn-v2-predictor-b695d4db9-f2t2n\" (UID: \"d1839a90-d614-43ed-883d-c5f37a1cebf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n"
Jun 15 06:37:15.890775 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.890757    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1839a90-d614-43ed-883d-c5f37a1cebf5-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-b695d4db9-f2t2n\" (UID: \"d1839a90-d614-43ed-883d-c5f37a1cebf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n"
Jun 15 06:37:15.890856 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.890840    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1839a90-d614-43ed-883d-c5f37a1cebf5-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-b695d4db9-f2t2n\" (UID: \"d1839a90-d614-43ed-883d-c5f37a1cebf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n"
Jun 15 06:37:15.891203 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.891170    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1839a90-d614-43ed-883d-c5f37a1cebf5-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-b695d4db9-f2t2n\" (UID: \"d1839a90-d614-43ed-883d-c5f37a1cebf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n"
Jun 15 06:37:15.891340 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.891256    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1839a90-d614-43ed-883d-c5f37a1cebf5-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-b695d4db9-f2t2n\" (UID: \"d1839a90-d614-43ed-883d-c5f37a1cebf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n"
Jun 15 06:37:15.891455 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.891351    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1839a90-d614-43ed-883d-c5f37a1cebf5-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-b695d4db9-f2t2n\" (UID: \"d1839a90-d614-43ed-883d-c5f37a1cebf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n"
Jun 15 06:37:15.892957 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.892935    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1839a90-d614-43ed-883d-c5f37a1cebf5-proxy-tls\") pod \"isvc-sklearn-v2-predictor-b695d4db9-f2t2n\" (UID: \"d1839a90-d614-43ed-883d-c5f37a1cebf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n"
Jun 15 06:37:15.899604 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.899577    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6c28\" (UniqueName: \"kubernetes.io/projected/d1839a90-d614-43ed-883d-c5f37a1cebf5-kube-api-access-c6c28\") pod \"isvc-sklearn-v2-predictor-b695d4db9-f2t2n\" (UID: \"d1839a90-d614-43ed-883d-c5f37a1cebf5\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n"
Jun 15 06:37:15.964459 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:15.964428    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n"
Jun 15 06:37:16.095756 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:16.095726    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n"]
Jun 15 06:37:16.097397 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:37:16.097343    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1839a90_d614_43ed_883d_c5f37a1cebf5.slice/crio-4e39d5cb8f15c0da54589d5917211303da513ec7d98f0dc520392adf317e9188 WatchSource:0}: Error finding container 4e39d5cb8f15c0da54589d5917211303da513ec7d98f0dc520392adf317e9188: Status 404 returned error can't find the container with id 4e39d5cb8f15c0da54589d5917211303da513ec7d98f0dc520392adf317e9188
Jun 15 06:37:16.378422 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:16.378387    2572 generic.go:356] "Generic (PLEG): container finished" podID="4b5e9f23-8aa8-45e7-b7b5-36c72e99c800" containerID="1f7ec8643c1562d3ee625a9e1d96cc13048e2f35506f7f3e6d777cfefcb82ea5" exitCode=2
Jun 15 06:37:16.378603 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:16.378459    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w" event={"ID":"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800","Type":"ContainerDied","Data":"1f7ec8643c1562d3ee625a9e1d96cc13048e2f35506f7f3e6d777cfefcb82ea5"}
Jun 15 06:37:16.379856 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:16.379835    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n" event={"ID":"d1839a90-d614-43ed-883d-c5f37a1cebf5","Type":"ContainerStarted","Data":"64df02d0c0e37fb9e86c76f261fb8ab56a8d0dd3ef00ac1ad93ff540b6eaa6e5"}
Jun 15 06:37:16.379994 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:16.379889    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n" event={"ID":"d1839a90-d614-43ed-883d-c5f37a1cebf5","Type":"ContainerStarted","Data":"4e39d5cb8f15c0da54589d5917211303da513ec7d98f0dc520392adf317e9188"}
Jun 15 06:37:20.137073 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:20.137027    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w" podUID="4b5e9f23-8aa8-45e7-b7b5-36c72e99c800" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Jun 15 06:37:20.142204 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:20.142174    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w" podUID="4b5e9f23-8aa8-45e7-b7b5-36c72e99c800" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.53:8080/v2/models/isvc-sklearn-v2-runtime/ready\": dial tcp 10.134.0.53:8080: connect: connection refused"
Jun 15 06:37:20.401573 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:20.401543    2572 generic.go:356] "Generic (PLEG): container finished" podID="d1839a90-d614-43ed-883d-c5f37a1cebf5" containerID="64df02d0c0e37fb9e86c76f261fb8ab56a8d0dd3ef00ac1ad93ff540b6eaa6e5" exitCode=0
Jun 15 06:37:20.401767 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:20.401620    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n" event={"ID":"d1839a90-d614-43ed-883d-c5f37a1cebf5","Type":"ContainerDied","Data":"64df02d0c0e37fb9e86c76f261fb8ab56a8d0dd3ef00ac1ad93ff540b6eaa6e5"}
Jun 15 06:37:21.408286 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:21.408247    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n" event={"ID":"d1839a90-d614-43ed-883d-c5f37a1cebf5","Type":"ContainerStarted","Data":"3a4d28d5030a2f31cb4e31fe8741fe44b0a7d9e6f909a387d7bafce50e30d65a"}
Jun 15 06:37:21.408713 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:21.408313    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n" event={"ID":"d1839a90-d614-43ed-883d-c5f37a1cebf5","Type":"ContainerStarted","Data":"f5c8135463984a185e2bbe634a372236f123fa75bdfcf9cfe2b3ab8be02fad86"}
Jun 15 06:37:21.408713 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:21.408635    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n"
Jun 15 06:37:21.429263 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:21.429211    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n" podStartSLOduration=2.693008161 podStartE2EDuration="6.429197138s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.736188977s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:37:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:37:21.427994127 +0000 UTC m=+2956.576174147" watchObservedRunningTime="2026-06-15 06:37:21.429197138 +0000 UTC m=+2956.577377134"
Jun 15 06:37:22.413230 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:22.413198    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n"
Jun 15 06:37:22.414201 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:22.414176    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n" podUID="d1839a90-d614-43ed-883d-c5f37a1cebf5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jun 15 06:37:23.018010 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.017986    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"
Jun 15 06:37:23.152268 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.152234    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-proxy-tls\") pod \"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800\" (UID: \"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800\") "
Jun 15 06:37:23.152462 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.152284    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-kube-rbac-proxy-sar-config\") pod \"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800\" (UID: \"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800\") "
Jun 15 06:37:23.152462 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.152321    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-kserve-provision-location\") pod \"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800\" (UID: \"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800\") "
Jun 15 06:37:23.152462 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.152395    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-cabundle-cert\") pod \"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800\" (UID: \"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800\") "
Jun 15 06:37:23.152462 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.152424    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-65lcv\" (UniqueName: \"kubernetes.io/projected/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-kube-api-access-65lcv\") pod \"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800\" (UID: \"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800\") "
Jun 15 06:37:23.152703 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.152677    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4b5e9f23-8aa8-45e7-b7b5-36c72e99c800" (UID: "4b5e9f23-8aa8-45e7-b7b5-36c72e99c800"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:37:23.152772 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.152706    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4b5e9f23-8aa8-45e7-b7b5-36c72e99c800" (UID: "4b5e9f23-8aa8-45e7-b7b5-36c72e99c800"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:37:23.152772 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.152729    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4b5e9f23-8aa8-45e7-b7b5-36c72e99c800" (UID: "4b5e9f23-8aa8-45e7-b7b5-36c72e99c800"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:37:23.152859 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.152812    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:37:23.152859 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.152826    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:37:23.152859 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.152836    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:37:23.154295 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.154273    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4b5e9f23-8aa8-45e7-b7b5-36c72e99c800" (UID: "4b5e9f23-8aa8-45e7-b7b5-36c72e99c800"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:37:23.154445 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.154423    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-kube-api-access-65lcv" (OuterVolumeSpecName: "kube-api-access-65lcv") pod "4b5e9f23-8aa8-45e7-b7b5-36c72e99c800" (UID: "4b5e9f23-8aa8-45e7-b7b5-36c72e99c800"). InnerVolumeSpecName "kube-api-access-65lcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:37:23.253795 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.253759    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:37:23.253795 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.253790    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65lcv\" (UniqueName: \"kubernetes.io/projected/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800-kube-api-access-65lcv\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:37:23.419178 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.419100    2572 generic.go:356] "Generic (PLEG): container finished" podID="4b5e9f23-8aa8-45e7-b7b5-36c72e99c800" containerID="32b012570a6c6984ab02abe9c1899366963748eea9021080ea5336c72dcb5963" exitCode=0
Jun 15 06:37:23.419178 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.419156    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"
Jun 15 06:37:23.419668 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.419175    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w" event={"ID":"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800","Type":"ContainerDied","Data":"32b012570a6c6984ab02abe9c1899366963748eea9021080ea5336c72dcb5963"}
Jun 15 06:37:23.419668 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.419227    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w" event={"ID":"4b5e9f23-8aa8-45e7-b7b5-36c72e99c800","Type":"ContainerDied","Data":"5b584d8b37440377f357463e99ff2aeb8e03c5bdbc1da5995ebd828c1a61b381"}
Jun 15 06:37:23.419668 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.419253    2572 scope.go:117] "RemoveContainer" containerID="1f7ec8643c1562d3ee625a9e1d96cc13048e2f35506f7f3e6d777cfefcb82ea5"
Jun 15 06:37:23.419668 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.419636    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n" podUID="d1839a90-d614-43ed-883d-c5f37a1cebf5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jun 15 06:37:23.428043 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.428025    2572 scope.go:117] "RemoveContainer" containerID="32b012570a6c6984ab02abe9c1899366963748eea9021080ea5336c72dcb5963"
Jun 15 06:37:23.435277 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.435259    2572 scope.go:117] "RemoveContainer" containerID="804963d63b1837a99b7f28f87568bf14925fa3c79d79296c1782d74a78f3e805"
Jun 15 06:37:23.437755 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.437738    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"]
Jun 15 06:37:23.442312 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.442290    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-runtime-predictor-5f69c7d498-vcl5w"]
Jun 15 06:37:23.444858 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.444844    2572 scope.go:117] "RemoveContainer" containerID="1f7ec8643c1562d3ee625a9e1d96cc13048e2f35506f7f3e6d777cfefcb82ea5"
Jun 15 06:37:23.445119 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:37:23.445099    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f7ec8643c1562d3ee625a9e1d96cc13048e2f35506f7f3e6d777cfefcb82ea5\": container with ID starting with 1f7ec8643c1562d3ee625a9e1d96cc13048e2f35506f7f3e6d777cfefcb82ea5 not found: ID does not exist" containerID="1f7ec8643c1562d3ee625a9e1d96cc13048e2f35506f7f3e6d777cfefcb82ea5"
Jun 15 06:37:23.445184 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.445131    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f7ec8643c1562d3ee625a9e1d96cc13048e2f35506f7f3e6d777cfefcb82ea5"} err="failed to get container status \"1f7ec8643c1562d3ee625a9e1d96cc13048e2f35506f7f3e6d777cfefcb82ea5\": rpc error: code = NotFound desc = could not find container \"1f7ec8643c1562d3ee625a9e1d96cc13048e2f35506f7f3e6d777cfefcb82ea5\": container with ID starting with 1f7ec8643c1562d3ee625a9e1d96cc13048e2f35506f7f3e6d777cfefcb82ea5 not found: ID does not exist"
Jun 15 06:37:23.445184 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.445148    2572 scope.go:117] "RemoveContainer" containerID="32b012570a6c6984ab02abe9c1899366963748eea9021080ea5336c72dcb5963"
Jun 15 06:37:23.445413 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:37:23.445397    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32b012570a6c6984ab02abe9c1899366963748eea9021080ea5336c72dcb5963\": container with ID starting with 32b012570a6c6984ab02abe9c1899366963748eea9021080ea5336c72dcb5963 not found: ID does not exist" containerID="32b012570a6c6984ab02abe9c1899366963748eea9021080ea5336c72dcb5963"
Jun 15 06:37:23.445461 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.445419    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32b012570a6c6984ab02abe9c1899366963748eea9021080ea5336c72dcb5963"} err="failed to get container status \"32b012570a6c6984ab02abe9c1899366963748eea9021080ea5336c72dcb5963\": rpc error: code = NotFound desc = could not find container \"32b012570a6c6984ab02abe9c1899366963748eea9021080ea5336c72dcb5963\": container with ID starting with 32b012570a6c6984ab02abe9c1899366963748eea9021080ea5336c72dcb5963 not found: ID does not exist"
Jun 15 06:37:23.445461 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.445432    2572 scope.go:117] "RemoveContainer" containerID="804963d63b1837a99b7f28f87568bf14925fa3c79d79296c1782d74a78f3e805"
Jun 15 06:37:23.445666 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:37:23.445650    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"804963d63b1837a99b7f28f87568bf14925fa3c79d79296c1782d74a78f3e805\": container with ID starting with 804963d63b1837a99b7f28f87568bf14925fa3c79d79296c1782d74a78f3e805 not found: ID does not exist" containerID="804963d63b1837a99b7f28f87568bf14925fa3c79d79296c1782d74a78f3e805"
Jun 15 06:37:23.445707 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:23.445669    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"804963d63b1837a99b7f28f87568bf14925fa3c79d79296c1782d74a78f3e805"} err="failed to get container status \"804963d63b1837a99b7f28f87568bf14925fa3c79d79296c1782d74a78f3e805\": rpc error: code = NotFound desc = could not find container \"804963d63b1837a99b7f28f87568bf14925fa3c79d79296c1782d74a78f3e805\": container with ID starting with 804963d63b1837a99b7f28f87568bf14925fa3c79d79296c1782d74a78f3e805 not found: ID does not exist"
Jun 15 06:37:25.323701 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:25.323668    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b5e9f23-8aa8-45e7-b7b5-36c72e99c800" path="/var/lib/kubelet/pods/4b5e9f23-8aa8-45e7-b7b5-36c72e99c800/volumes"
Jun 15 06:37:28.424108 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:28.424083    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n"
Jun 15 06:37:28.424583 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:28.424555    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n" podUID="d1839a90-d614-43ed-883d-c5f37a1cebf5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jun 15 06:37:38.425490 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:38.425452    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n" podUID="d1839a90-d614-43ed-883d-c5f37a1cebf5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jun 15 06:37:48.424524 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:48.424485    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n" podUID="d1839a90-d614-43ed-883d-c5f37a1cebf5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jun 15 06:37:58.425073 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:37:58.425032    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n" podUID="d1839a90-d614-43ed-883d-c5f37a1cebf5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jun 15 06:38:05.540917 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:05.540879    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 06:38:05.560246 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:05.560225    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 06:38:08.424990 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:08.424946    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n" podUID="d1839a90-d614-43ed-883d-c5f37a1cebf5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jun 15 06:38:18.424505 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:18.424464    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n" podUID="d1839a90-d614-43ed-883d-c5f37a1cebf5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jun 15 06:38:28.425376 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:28.425329    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n" podUID="d1839a90-d614-43ed-883d-c5f37a1cebf5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jun 15 06:38:31.319467 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:31.319417    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n" podUID="d1839a90-d614-43ed-883d-c5f37a1cebf5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jun 15 06:38:41.323316 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:41.323289    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n"
Jun 15 06:38:45.898152 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:45.898043    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n"]
Jun 15 06:38:45.898637 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:45.898354    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n" podUID="d1839a90-d614-43ed-883d-c5f37a1cebf5" containerName="kserve-container" containerID="cri-o://f5c8135463984a185e2bbe634a372236f123fa75bdfcf9cfe2b3ab8be02fad86" gracePeriod=30
Jun 15 06:38:45.898637 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:45.898454    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n" podUID="d1839a90-d614-43ed-883d-c5f37a1cebf5" containerName="kube-rbac-proxy" containerID="cri-o://3a4d28d5030a2f31cb4e31fe8741fe44b0a7d9e6f909a387d7bafce50e30d65a" gracePeriod=30
Jun 15 06:38:46.007805 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.007772    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88"]
Jun 15 06:38:46.008168 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.008146    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b5e9f23-8aa8-45e7-b7b5-36c72e99c800" containerName="kube-rbac-proxy"
Jun 15 06:38:46.008168 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.008165    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b5e9f23-8aa8-45e7-b7b5-36c72e99c800" containerName="kube-rbac-proxy"
Jun 15 06:38:46.008337 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.008192    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b5e9f23-8aa8-45e7-b7b5-36c72e99c800" containerName="kserve-container"
Jun 15 06:38:46.008337 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.008197    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b5e9f23-8aa8-45e7-b7b5-36c72e99c800" containerName="kserve-container"
Jun 15 06:38:46.008337 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.008206    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b5e9f23-8aa8-45e7-b7b5-36c72e99c800" containerName="storage-initializer"
Jun 15 06:38:46.008337 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.008212    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b5e9f23-8aa8-45e7-b7b5-36c72e99c800" containerName="storage-initializer"
Jun 15 06:38:46.008337 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.008267    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b5e9f23-8aa8-45e7-b7b5-36c72e99c800" containerName="kserve-container"
Jun 15 06:38:46.008337 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.008274    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b5e9f23-8aa8-45e7-b7b5-36c72e99c800" containerName="kube-rbac-proxy"
Jun 15 06:38:46.011603 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.011583    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88"
Jun 15 06:38:46.013804 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.013781    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-predictor-serving-cert\""
Jun 15 06:38:46.013902 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.013806    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-mixed-kube-rbac-proxy-sar-config\""
Jun 15 06:38:46.023244 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.023222    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88"]
Jun 15 06:38:46.103793 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.103762    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6tkb\" (UniqueName: \"kubernetes.io/projected/624e4196-936d-4368-8804-201e7457e398-kube-api-access-g6tkb\") pod \"isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88\" (UID: \"624e4196-936d-4368-8804-201e7457e398\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88"
Jun 15 06:38:46.103958 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.103800    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/624e4196-936d-4368-8804-201e7457e398-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88\" (UID: \"624e4196-936d-4368-8804-201e7457e398\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88"
Jun 15 06:38:46.103958 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.103906    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/624e4196-936d-4368-8804-201e7457e398-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88\" (UID: \"624e4196-936d-4368-8804-201e7457e398\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88"
Jun 15 06:38:46.103958 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.103939    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/624e4196-936d-4368-8804-201e7457e398-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88\" (UID: \"624e4196-936d-4368-8804-201e7457e398\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88"
Jun 15 06:38:46.104111 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.103976    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/624e4196-936d-4368-8804-201e7457e398-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88\" (UID: \"624e4196-936d-4368-8804-201e7457e398\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88"
Jun 15 06:38:46.205299 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.205211    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/624e4196-936d-4368-8804-201e7457e398-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88\" (UID: \"624e4196-936d-4368-8804-201e7457e398\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88"
Jun 15 06:38:46.205491 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.205378    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6tkb\" (UniqueName: \"kubernetes.io/projected/624e4196-936d-4368-8804-201e7457e398-kube-api-access-g6tkb\") pod \"isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88\" (UID: \"624e4196-936d-4368-8804-201e7457e398\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88"
Jun 15 06:38:46.205491 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.205419    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/624e4196-936d-4368-8804-201e7457e398-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88\" (UID: \"624e4196-936d-4368-8804-201e7457e398\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88"
Jun 15 06:38:46.205609 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.205489    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/624e4196-936d-4368-8804-201e7457e398-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88\" (UID: \"624e4196-936d-4368-8804-201e7457e398\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88"
Jun 15 06:38:46.205609 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.205514    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/624e4196-936d-4368-8804-201e7457e398-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88\" (UID: \"624e4196-936d-4368-8804-201e7457e398\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88"
Jun 15 06:38:46.205833 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.205812    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/624e4196-936d-4368-8804-201e7457e398-kserve-provision-location\") pod \"isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88\" (UID: \"624e4196-936d-4368-8804-201e7457e398\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88"
Jun 15 06:38:46.205976 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.205955    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/624e4196-936d-4368-8804-201e7457e398-cabundle-cert\") pod \"isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88\" (UID: \"624e4196-936d-4368-8804-201e7457e398\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88"
Jun 15 06:38:46.206151 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.206123    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/624e4196-936d-4368-8804-201e7457e398-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88\" (UID: \"624e4196-936d-4368-8804-201e7457e398\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88"
Jun 15 06:38:46.207887 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.207870    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/624e4196-936d-4368-8804-201e7457e398-proxy-tls\") pod \"isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88\" (UID: \"624e4196-936d-4368-8804-201e7457e398\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88"
Jun 15 06:38:46.215886 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.215864    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6tkb\" (UniqueName: \"kubernetes.io/projected/624e4196-936d-4368-8804-201e7457e398-kube-api-access-g6tkb\") pod \"isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88\" (UID: \"624e4196-936d-4368-8804-201e7457e398\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88"
Jun 15 06:38:46.322513 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.322478    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88"
Jun 15 06:38:46.457747 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.457724    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88"]
Jun 15 06:38:46.459756 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:38:46.459725    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod624e4196_936d_4368_8804_201e7457e398.slice/crio-3cb492a127206bc181b79b0d951896c3731b131622979d8815928326b97b3595 WatchSource:0}: Error finding container 3cb492a127206bc181b79b0d951896c3731b131622979d8815928326b97b3595: Status 404 returned error can't find the container with id 3cb492a127206bc181b79b0d951896c3731b131622979d8815928326b97b3595
Jun 15 06:38:46.814669 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.814590    2572 generic.go:356] "Generic (PLEG): container finished" podID="d1839a90-d614-43ed-883d-c5f37a1cebf5" containerID="3a4d28d5030a2f31cb4e31fe8741fe44b0a7d9e6f909a387d7bafce50e30d65a" exitCode=2
Jun 15 06:38:46.814815 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.814675    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n" event={"ID":"d1839a90-d614-43ed-883d-c5f37a1cebf5","Type":"ContainerDied","Data":"3a4d28d5030a2f31cb4e31fe8741fe44b0a7d9e6f909a387d7bafce50e30d65a"}
Jun 15 06:38:46.816215 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.816192    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88" event={"ID":"624e4196-936d-4368-8804-201e7457e398","Type":"ContainerStarted","Data":"cef40aba2d869cd07f34d851f8fa6d69d1ab14cf76f2d28f084c06a922d283b6"}
Jun 15 06:38:46.816374 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:46.816269    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88" event={"ID":"624e4196-936d-4368-8804-201e7457e398","Type":"ContainerStarted","Data":"3cb492a127206bc181b79b0d951896c3731b131622979d8815928326b97b3595"}
Jun 15 06:38:48.419581 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:48.419532    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n" podUID="d1839a90-d614-43ed-883d-c5f37a1cebf5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Jun 15 06:38:50.645921 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.645896    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n"
Jun 15 06:38:50.746423 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.746303    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1839a90-d614-43ed-883d-c5f37a1cebf5-cabundle-cert\") pod \"d1839a90-d614-43ed-883d-c5f37a1cebf5\" (UID: \"d1839a90-d614-43ed-883d-c5f37a1cebf5\") "
Jun 15 06:38:50.746423 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.746417    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6c28\" (UniqueName: \"kubernetes.io/projected/d1839a90-d614-43ed-883d-c5f37a1cebf5-kube-api-access-c6c28\") pod \"d1839a90-d614-43ed-883d-c5f37a1cebf5\" (UID: \"d1839a90-d614-43ed-883d-c5f37a1cebf5\") "
Jun 15 06:38:50.746665 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.746474    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1839a90-d614-43ed-883d-c5f37a1cebf5-kube-rbac-proxy-sar-config\") pod \"d1839a90-d614-43ed-883d-c5f37a1cebf5\" (UID: \"d1839a90-d614-43ed-883d-c5f37a1cebf5\") "
Jun 15 06:38:50.746665 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.746495    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1839a90-d614-43ed-883d-c5f37a1cebf5-proxy-tls\") pod \"d1839a90-d614-43ed-883d-c5f37a1cebf5\" (UID: \"d1839a90-d614-43ed-883d-c5f37a1cebf5\") "
Jun 15 06:38:50.746665 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.746518    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1839a90-d614-43ed-883d-c5f37a1cebf5-kserve-provision-location\") pod \"d1839a90-d614-43ed-883d-c5f37a1cebf5\" (UID: \"d1839a90-d614-43ed-883d-c5f37a1cebf5\") "
Jun 15 06:38:50.746813 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.746775    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1839a90-d614-43ed-883d-c5f37a1cebf5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d1839a90-d614-43ed-883d-c5f37a1cebf5" (UID: "d1839a90-d614-43ed-883d-c5f37a1cebf5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:38:50.746904 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.746879    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1839a90-d614-43ed-883d-c5f37a1cebf5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d1839a90-d614-43ed-883d-c5f37a1cebf5" (UID: "d1839a90-d614-43ed-883d-c5f37a1cebf5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:38:50.746964 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.746888    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1839a90-d614-43ed-883d-c5f37a1cebf5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d1839a90-d614-43ed-883d-c5f37a1cebf5" (UID: "d1839a90-d614-43ed-883d-c5f37a1cebf5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:38:50.748613 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.748594    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1839a90-d614-43ed-883d-c5f37a1cebf5-kube-api-access-c6c28" (OuterVolumeSpecName: "kube-api-access-c6c28") pod "d1839a90-d614-43ed-883d-c5f37a1cebf5" (UID: "d1839a90-d614-43ed-883d-c5f37a1cebf5"). InnerVolumeSpecName "kube-api-access-c6c28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:38:50.748716 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.748696    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1839a90-d614-43ed-883d-c5f37a1cebf5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d1839a90-d614-43ed-883d-c5f37a1cebf5" (UID: "d1839a90-d614-43ed-883d-c5f37a1cebf5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:38:50.835640 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.835607    2572 generic.go:356] "Generic (PLEG): container finished" podID="624e4196-936d-4368-8804-201e7457e398" containerID="cef40aba2d869cd07f34d851f8fa6d69d1ab14cf76f2d28f084c06a922d283b6" exitCode=0
Jun 15 06:38:50.835786 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.835682    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88" event={"ID":"624e4196-936d-4368-8804-201e7457e398","Type":"ContainerDied","Data":"cef40aba2d869cd07f34d851f8fa6d69d1ab14cf76f2d28f084c06a922d283b6"}
Jun 15 06:38:50.837783 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.837752    2572 generic.go:356] "Generic (PLEG): container finished" podID="d1839a90-d614-43ed-883d-c5f37a1cebf5" containerID="f5c8135463984a185e2bbe634a372236f123fa75bdfcf9cfe2b3ab8be02fad86" exitCode=0
Jun 15 06:38:50.837891 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.837809    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n"
Jun 15 06:38:50.837891 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.837831    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n" event={"ID":"d1839a90-d614-43ed-883d-c5f37a1cebf5","Type":"ContainerDied","Data":"f5c8135463984a185e2bbe634a372236f123fa75bdfcf9cfe2b3ab8be02fad86"}
Jun 15 06:38:50.837891 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.837884    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n" event={"ID":"d1839a90-d614-43ed-883d-c5f37a1cebf5","Type":"ContainerDied","Data":"4e39d5cb8f15c0da54589d5917211303da513ec7d98f0dc520392adf317e9188"}
Jun 15 06:38:50.838014 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.837915    2572 scope.go:117] "RemoveContainer" containerID="3a4d28d5030a2f31cb4e31fe8741fe44b0a7d9e6f909a387d7bafce50e30d65a"
Jun 15 06:38:50.847584 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.847561    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d1839a90-d614-43ed-883d-c5f37a1cebf5-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:38:50.847673 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.847586    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d1839a90-d614-43ed-883d-c5f37a1cebf5-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:38:50.847673 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.847602    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6c28\" (UniqueName: \"kubernetes.io/projected/d1839a90-d614-43ed-883d-c5f37a1cebf5-kube-api-access-c6c28\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:38:50.847673 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.847614    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1839a90-d614-43ed-883d-c5f37a1cebf5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:38:50.847673 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.847627    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1839a90-d614-43ed-883d-c5f37a1cebf5-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:38:50.849125 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.849113    2572 scope.go:117] "RemoveContainer" containerID="f5c8135463984a185e2bbe634a372236f123fa75bdfcf9cfe2b3ab8be02fad86"
Jun 15 06:38:50.861417 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.861398    2572 scope.go:117] "RemoveContainer" containerID="64df02d0c0e37fb9e86c76f261fb8ab56a8d0dd3ef00ac1ad93ff540b6eaa6e5"
Jun 15 06:38:50.870979 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.870955    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n"]
Jun 15 06:38:50.873969 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.873949    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-b695d4db9-f2t2n"]
Jun 15 06:38:50.877839 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.877820    2572 scope.go:117] "RemoveContainer" containerID="3a4d28d5030a2f31cb4e31fe8741fe44b0a7d9e6f909a387d7bafce50e30d65a"
Jun 15 06:38:50.878240 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:38:50.878218    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a4d28d5030a2f31cb4e31fe8741fe44b0a7d9e6f909a387d7bafce50e30d65a\": container with ID starting with 3a4d28d5030a2f31cb4e31fe8741fe44b0a7d9e6f909a387d7bafce50e30d65a not found: ID does not exist" containerID="3a4d28d5030a2f31cb4e31fe8741fe44b0a7d9e6f909a387d7bafce50e30d65a"
Jun 15 06:38:50.878322 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.878249    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a4d28d5030a2f31cb4e31fe8741fe44b0a7d9e6f909a387d7bafce50e30d65a"} err="failed to get container status \"3a4d28d5030a2f31cb4e31fe8741fe44b0a7d9e6f909a387d7bafce50e30d65a\": rpc error: code = NotFound desc = could not find container \"3a4d28d5030a2f31cb4e31fe8741fe44b0a7d9e6f909a387d7bafce50e30d65a\": container with ID starting with 3a4d28d5030a2f31cb4e31fe8741fe44b0a7d9e6f909a387d7bafce50e30d65a not found: ID does not exist"
Jun 15 06:38:50.878322 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.878268    2572 scope.go:117] "RemoveContainer" containerID="f5c8135463984a185e2bbe634a372236f123fa75bdfcf9cfe2b3ab8be02fad86"
Jun 15 06:38:50.878546 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:38:50.878531    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5c8135463984a185e2bbe634a372236f123fa75bdfcf9cfe2b3ab8be02fad86\": container with ID starting with f5c8135463984a185e2bbe634a372236f123fa75bdfcf9cfe2b3ab8be02fad86 not found: ID does not exist" containerID="f5c8135463984a185e2bbe634a372236f123fa75bdfcf9cfe2b3ab8be02fad86"
Jun 15 06:38:50.878591 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.878551    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c8135463984a185e2bbe634a372236f123fa75bdfcf9cfe2b3ab8be02fad86"} err="failed to get container status \"f5c8135463984a185e2bbe634a372236f123fa75bdfcf9cfe2b3ab8be02fad86\": rpc error: code = NotFound desc = could not find container \"f5c8135463984a185e2bbe634a372236f123fa75bdfcf9cfe2b3ab8be02fad86\": container with ID starting with f5c8135463984a185e2bbe634a372236f123fa75bdfcf9cfe2b3ab8be02fad86 not found: ID does not exist"
Jun 15 06:38:50.878591 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.878565    2572 scope.go:117] "RemoveContainer" containerID="64df02d0c0e37fb9e86c76f261fb8ab56a8d0dd3ef00ac1ad93ff540b6eaa6e5"
Jun 15 06:38:50.878842 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:38:50.878825    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64df02d0c0e37fb9e86c76f261fb8ab56a8d0dd3ef00ac1ad93ff540b6eaa6e5\": container with ID starting with 64df02d0c0e37fb9e86c76f261fb8ab56a8d0dd3ef00ac1ad93ff540b6eaa6e5 not found: ID does not exist" containerID="64df02d0c0e37fb9e86c76f261fb8ab56a8d0dd3ef00ac1ad93ff540b6eaa6e5"
Jun 15 06:38:50.878929 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:50.878856    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64df02d0c0e37fb9e86c76f261fb8ab56a8d0dd3ef00ac1ad93ff540b6eaa6e5"} err="failed to get container status \"64df02d0c0e37fb9e86c76f261fb8ab56a8d0dd3ef00ac1ad93ff540b6eaa6e5\": rpc error: code = NotFound desc = could not find container \"64df02d0c0e37fb9e86c76f261fb8ab56a8d0dd3ef00ac1ad93ff540b6eaa6e5\": container with ID starting with 64df02d0c0e37fb9e86c76f261fb8ab56a8d0dd3ef00ac1ad93ff540b6eaa6e5 not found: ID does not exist"
Jun 15 06:38:51.324769 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:51.324734    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1839a90-d614-43ed-883d-c5f37a1cebf5" path="/var/lib/kubelet/pods/d1839a90-d614-43ed-883d-c5f37a1cebf5/volumes"
Jun 15 06:38:51.845010 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:51.844978    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88" event={"ID":"624e4196-936d-4368-8804-201e7457e398","Type":"ContainerStarted","Data":"baa35230fe8b30924a4fd3b1631013acca8149eff6e3e60eecc9af13fbb084e8"}
Jun 15 06:38:51.845538 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:51.845043    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88" event={"ID":"624e4196-936d-4368-8804-201e7457e398","Type":"ContainerStarted","Data":"cc7bf04a9fde7dda767a20a883514d0009d450936ff11d9d24fcf7f523c923b7"}
Jun 15 06:38:51.845538 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:51.845228    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88"
Jun 15 06:38:51.866508 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:51.866461    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88" podStartSLOduration=2.890887168 podStartE2EDuration="6.866446357s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.975559189s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:38:51.864732719 +0000 UTC m=+3047.012912714" watchObservedRunningTime="2026-06-15 06:38:51.866446357 +0000 UTC m=+3047.014626354"
Jun 15 06:38:52.850520 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:52.850491    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88"
Jun 15 06:38:52.851634 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:52.851612    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88" podUID="624e4196-936d-4368-8804-201e7457e398" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jun 15 06:38:53.855099 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:53.855059    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88" podUID="624e4196-936d-4368-8804-201e7457e398" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jun 15 06:38:58.859497 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:58.859470    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88"
Jun 15 06:38:58.859995 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:38:58.859969    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88" podUID="624e4196-936d-4368-8804-201e7457e398" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jun 15 06:39:08.860020 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:39:08.859979    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88" podUID="624e4196-936d-4368-8804-201e7457e398" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jun 15 06:39:18.859906 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:39:18.859865    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88" podUID="624e4196-936d-4368-8804-201e7457e398" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jun 15 06:39:28.860616 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:39:28.860579    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88" podUID="624e4196-936d-4368-8804-201e7457e398" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jun 15 06:39:38.860448 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:39:38.860409    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88" podUID="624e4196-936d-4368-8804-201e7457e398" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jun 15 06:39:48.860876 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:39:48.860841    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88" podUID="624e4196-936d-4368-8804-201e7457e398" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jun 15 06:39:58.860787 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:39:58.860743    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88" podUID="624e4196-936d-4368-8804-201e7457e398" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jun 15 06:40:08.860562 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:08.860529    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88"
Jun 15 06:40:16.203567 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:16.203474    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88"]
Jun 15 06:40:16.204040 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:16.203802    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88" podUID="624e4196-936d-4368-8804-201e7457e398" containerName="kube-rbac-proxy" containerID="cri-o://baa35230fe8b30924a4fd3b1631013acca8149eff6e3e60eecc9af13fbb084e8" gracePeriod=30
Jun 15 06:40:16.204040 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:16.203761    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88" podUID="624e4196-936d-4368-8804-201e7457e398" containerName="kserve-container" containerID="cri-o://cc7bf04a9fde7dda767a20a883514d0009d450936ff11d9d24fcf7f523c923b7" gracePeriod=30
Jun 15 06:40:17.248154 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:17.248122    2572 generic.go:356] "Generic (PLEG): container finished" podID="624e4196-936d-4368-8804-201e7457e398" containerID="baa35230fe8b30924a4fd3b1631013acca8149eff6e3e60eecc9af13fbb084e8" exitCode=2
Jun 15 06:40:17.248154 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:17.248160    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88" event={"ID":"624e4196-936d-4368-8804-201e7457e398","Type":"ContainerDied","Data":"baa35230fe8b30924a4fd3b1631013acca8149eff6e3e60eecc9af13fbb084e8"}
Jun 15 06:40:18.856084 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:18.856038    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88" podUID="624e4196-936d-4368-8804-201e7457e398" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
Jun 15 06:40:18.860638 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:18.860612    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88" podUID="624e4196-936d-4368-8804-201e7457e398" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jun 15 06:40:20.955057 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:20.955035    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88"
Jun 15 06:40:21.036150 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.036062    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/624e4196-936d-4368-8804-201e7457e398-cabundle-cert\") pod \"624e4196-936d-4368-8804-201e7457e398\" (UID: \"624e4196-936d-4368-8804-201e7457e398\") "
Jun 15 06:40:21.036150 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.036102    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/624e4196-936d-4368-8804-201e7457e398-kserve-provision-location\") pod \"624e4196-936d-4368-8804-201e7457e398\" (UID: \"624e4196-936d-4368-8804-201e7457e398\") "
Jun 15 06:40:21.036150 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.036140    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/624e4196-936d-4368-8804-201e7457e398-proxy-tls\") pod \"624e4196-936d-4368-8804-201e7457e398\" (UID: \"624e4196-936d-4368-8804-201e7457e398\") "
Jun 15 06:40:21.036429 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.036170    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/624e4196-936d-4368-8804-201e7457e398-kube-rbac-proxy-sar-config\") pod \"624e4196-936d-4368-8804-201e7457e398\" (UID: \"624e4196-936d-4368-8804-201e7457e398\") "
Jun 15 06:40:21.036429 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.036209    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6tkb\" (UniqueName: \"kubernetes.io/projected/624e4196-936d-4368-8804-201e7457e398-kube-api-access-g6tkb\") pod \"624e4196-936d-4368-8804-201e7457e398\" (UID: \"624e4196-936d-4368-8804-201e7457e398\") "
Jun 15 06:40:21.036548 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.036501    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/624e4196-936d-4368-8804-201e7457e398-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "624e4196-936d-4368-8804-201e7457e398" (UID: "624e4196-936d-4368-8804-201e7457e398"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:40:21.036548 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.036502    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/624e4196-936d-4368-8804-201e7457e398-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "624e4196-936d-4368-8804-201e7457e398" (UID: "624e4196-936d-4368-8804-201e7457e398"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:40:21.036659 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.036551    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/624e4196-936d-4368-8804-201e7457e398-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "624e4196-936d-4368-8804-201e7457e398" (UID: "624e4196-936d-4368-8804-201e7457e398"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:40:21.038247 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.038230    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/624e4196-936d-4368-8804-201e7457e398-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "624e4196-936d-4368-8804-201e7457e398" (UID: "624e4196-936d-4368-8804-201e7457e398"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:40:21.038382 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.038347    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/624e4196-936d-4368-8804-201e7457e398-kube-api-access-g6tkb" (OuterVolumeSpecName: "kube-api-access-g6tkb") pod "624e4196-936d-4368-8804-201e7457e398" (UID: "624e4196-936d-4368-8804-201e7457e398"). InnerVolumeSpecName "kube-api-access-g6tkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:40:21.137666 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.137616    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6tkb\" (UniqueName: \"kubernetes.io/projected/624e4196-936d-4368-8804-201e7457e398-kube-api-access-g6tkb\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:40:21.137666 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.137659    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/624e4196-936d-4368-8804-201e7457e398-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:40:21.137666 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.137669    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/624e4196-936d-4368-8804-201e7457e398-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:40:21.137666 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.137678    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/624e4196-936d-4368-8804-201e7457e398-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:40:21.137666 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.137688    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/624e4196-936d-4368-8804-201e7457e398-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:40:21.266526 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.266493    2572 generic.go:356] "Generic (PLEG): container finished" podID="624e4196-936d-4368-8804-201e7457e398" containerID="cc7bf04a9fde7dda767a20a883514d0009d450936ff11d9d24fcf7f523c923b7" exitCode=0
Jun 15 06:40:21.266696 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.266551    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88"
Jun 15 06:40:21.266696 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.266582    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88" event={"ID":"624e4196-936d-4368-8804-201e7457e398","Type":"ContainerDied","Data":"cc7bf04a9fde7dda767a20a883514d0009d450936ff11d9d24fcf7f523c923b7"}
Jun 15 06:40:21.266696 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.266641    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88" event={"ID":"624e4196-936d-4368-8804-201e7457e398","Type":"ContainerDied","Data":"3cb492a127206bc181b79b0d951896c3731b131622979d8815928326b97b3595"}
Jun 15 06:40:21.266696 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.266668    2572 scope.go:117] "RemoveContainer" containerID="baa35230fe8b30924a4fd3b1631013acca8149eff6e3e60eecc9af13fbb084e8"
Jun 15 06:40:21.275431 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.275409    2572 scope.go:117] "RemoveContainer" containerID="cc7bf04a9fde7dda767a20a883514d0009d450936ff11d9d24fcf7f523c923b7"
Jun 15 06:40:21.282764 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.282748    2572 scope.go:117] "RemoveContainer" containerID="cef40aba2d869cd07f34d851f8fa6d69d1ab14cf76f2d28f084c06a922d283b6"
Jun 15 06:40:21.290592 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.290561    2572 scope.go:117] "RemoveContainer" containerID="baa35230fe8b30924a4fd3b1631013acca8149eff6e3e60eecc9af13fbb084e8"
Jun 15 06:40:21.290922 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:40:21.290901    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baa35230fe8b30924a4fd3b1631013acca8149eff6e3e60eecc9af13fbb084e8\": container with ID starting with baa35230fe8b30924a4fd3b1631013acca8149eff6e3e60eecc9af13fbb084e8 not found: ID does not exist" containerID="baa35230fe8b30924a4fd3b1631013acca8149eff6e3e60eecc9af13fbb084e8"
Jun 15 06:40:21.291012 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.290934    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baa35230fe8b30924a4fd3b1631013acca8149eff6e3e60eecc9af13fbb084e8"} err="failed to get container status \"baa35230fe8b30924a4fd3b1631013acca8149eff6e3e60eecc9af13fbb084e8\": rpc error: code = NotFound desc = could not find container \"baa35230fe8b30924a4fd3b1631013acca8149eff6e3e60eecc9af13fbb084e8\": container with ID starting with baa35230fe8b30924a4fd3b1631013acca8149eff6e3e60eecc9af13fbb084e8 not found: ID does not exist"
Jun 15 06:40:21.291012 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.290956    2572 scope.go:117] "RemoveContainer" containerID="cc7bf04a9fde7dda767a20a883514d0009d450936ff11d9d24fcf7f523c923b7"
Jun 15 06:40:21.291114 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.291101    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88"]
Jun 15 06:40:21.291290 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:40:21.291262    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc7bf04a9fde7dda767a20a883514d0009d450936ff11d9d24fcf7f523c923b7\": container with ID starting with cc7bf04a9fde7dda767a20a883514d0009d450936ff11d9d24fcf7f523c923b7 not found: ID does not exist" containerID="cc7bf04a9fde7dda767a20a883514d0009d450936ff11d9d24fcf7f523c923b7"
Jun 15 06:40:21.291340 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.291289    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc7bf04a9fde7dda767a20a883514d0009d450936ff11d9d24fcf7f523c923b7"} err="failed to get container status \"cc7bf04a9fde7dda767a20a883514d0009d450936ff11d9d24fcf7f523c923b7\": rpc error: code = NotFound desc = could not find container \"cc7bf04a9fde7dda767a20a883514d0009d450936ff11d9d24fcf7f523c923b7\": container with ID starting with cc7bf04a9fde7dda767a20a883514d0009d450936ff11d9d24fcf7f523c923b7 not found: ID does not exist"
Jun 15 06:40:21.291340 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.291306    2572 scope.go:117] "RemoveContainer" containerID="cef40aba2d869cd07f34d851f8fa6d69d1ab14cf76f2d28f084c06a922d283b6"
Jun 15 06:40:21.291549 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:40:21.291526    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cef40aba2d869cd07f34d851f8fa6d69d1ab14cf76f2d28f084c06a922d283b6\": container with ID starting with cef40aba2d869cd07f34d851f8fa6d69d1ab14cf76f2d28f084c06a922d283b6 not found: ID does not exist" containerID="cef40aba2d869cd07f34d851f8fa6d69d1ab14cf76f2d28f084c06a922d283b6"
Jun 15 06:40:21.291610 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.291559    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cef40aba2d869cd07f34d851f8fa6d69d1ab14cf76f2d28f084c06a922d283b6"} err="failed to get container status \"cef40aba2d869cd07f34d851f8fa6d69d1ab14cf76f2d28f084c06a922d283b6\": rpc error: code = NotFound desc = could not find container \"cef40aba2d869cd07f34d851f8fa6d69d1ab14cf76f2d28f084c06a922d283b6\": container with ID starting with cef40aba2d869cd07f34d851f8fa6d69d1ab14cf76f2d28f084c06a922d283b6 not found: ID does not exist"
Jun 15 06:40:21.295984 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.295965    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-mixed-predictor-7b87689dd8-c9v88"]
Jun 15 06:40:21.323210 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:40:21.323182    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="624e4196-936d-4368-8804-201e7457e398" path="/var/lib/kubelet/pods/624e4196-936d-4368-8804-201e7457e398/volumes"
Jun 15 06:43:05.569280 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:43:05.569251    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 06:43:05.593494 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:43:05.593472    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 06:45:33.162394 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.162342    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6"]
Jun 15 06:45:33.164868 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.162703    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1839a90-d614-43ed-883d-c5f37a1cebf5" containerName="kserve-container"
Jun 15 06:45:33.164868 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.162715    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1839a90-d614-43ed-883d-c5f37a1cebf5" containerName="kserve-container"
Jun 15 06:45:33.164868 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.162731    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1839a90-d614-43ed-883d-c5f37a1cebf5" containerName="storage-initializer"
Jun 15 06:45:33.164868 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.162736    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1839a90-d614-43ed-883d-c5f37a1cebf5" containerName="storage-initializer"
Jun 15 06:45:33.164868 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.162746    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="624e4196-936d-4368-8804-201e7457e398" containerName="kube-rbac-proxy"
Jun 15 06:45:33.164868 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.162752    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="624e4196-936d-4368-8804-201e7457e398" containerName="kube-rbac-proxy"
Jun 15 06:45:33.164868 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.162763    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1839a90-d614-43ed-883d-c5f37a1cebf5" containerName="kube-rbac-proxy"
Jun 15 06:45:33.164868 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.162768    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1839a90-d614-43ed-883d-c5f37a1cebf5" containerName="kube-rbac-proxy"
Jun 15 06:45:33.164868 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.162776    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="624e4196-936d-4368-8804-201e7457e398" containerName="kserve-container"
Jun 15 06:45:33.164868 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.162783    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="624e4196-936d-4368-8804-201e7457e398" containerName="kserve-container"
Jun 15 06:45:33.164868 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.162789    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="624e4196-936d-4368-8804-201e7457e398" containerName="storage-initializer"
Jun 15 06:45:33.164868 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.162794    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="624e4196-936d-4368-8804-201e7457e398" containerName="storage-initializer"
Jun 15 06:45:33.164868 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.162841    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="624e4196-936d-4368-8804-201e7457e398" containerName="kube-rbac-proxy"
Jun 15 06:45:33.164868 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.162851    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1839a90-d614-43ed-883d-c5f37a1cebf5" containerName="kube-rbac-proxy"
Jun 15 06:45:33.164868 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.162857    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="624e4196-936d-4368-8804-201e7457e398" containerName="kserve-container"
Jun 15 06:45:33.164868 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.162864    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1839a90-d614-43ed-883d-c5f37a1cebf5" containerName="kserve-container"
Jun 15 06:45:33.165990 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.165974    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6"
Jun 15 06:45:33.169582 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.169561    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jun 15 06:45:33.169796 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.169775    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jun 15 06:45:33.169875 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.169780    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-mlserver-predictor-serving-cert\""
Jun 15 06:45:33.169875 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.169785    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jun 15 06:45:33.169875 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.169853    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-q7vt7\""
Jun 15 06:45:33.170311 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.170297    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 15 06:45:33.173222 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.173201    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 15 06:45:33.183729 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.183708    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6"]
Jun 15 06:45:33.281178 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.281141    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaadbe2a-a6f9-4335-b134-3a27e52c6685-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6\" (UID: \"eaadbe2a-a6f9-4335-b134-3a27e52c6685\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6"
Jun 15 06:45:33.281385 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.281192    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eaadbe2a-a6f9-4335-b134-3a27e52c6685-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6\" (UID: \"eaadbe2a-a6f9-4335-b134-3a27e52c6685\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6"
Jun 15 06:45:33.281385 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.281224    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaadbe2a-a6f9-4335-b134-3a27e52c6685-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6\" (UID: \"eaadbe2a-a6f9-4335-b134-3a27e52c6685\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6"
Jun 15 06:45:33.281385 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.281268    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t2px\" (UniqueName: \"kubernetes.io/projected/eaadbe2a-a6f9-4335-b134-3a27e52c6685-kube-api-access-8t2px\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6\" (UID: \"eaadbe2a-a6f9-4335-b134-3a27e52c6685\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6"
Jun 15 06:45:33.281385 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.281322    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaadbe2a-a6f9-4335-b134-3a27e52c6685-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6\" (UID: \"eaadbe2a-a6f9-4335-b134-3a27e52c6685\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6"
Jun 15 06:45:33.382245 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.382207    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaadbe2a-a6f9-4335-b134-3a27e52c6685-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6\" (UID: \"eaadbe2a-a6f9-4335-b134-3a27e52c6685\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6"
Jun 15 06:45:33.382453 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.382286    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t2px\" (UniqueName: \"kubernetes.io/projected/eaadbe2a-a6f9-4335-b134-3a27e52c6685-kube-api-access-8t2px\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6\" (UID: \"eaadbe2a-a6f9-4335-b134-3a27e52c6685\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6"
Jun 15 06:45:33.382453 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.382343    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaadbe2a-a6f9-4335-b134-3a27e52c6685-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6\" (UID: \"eaadbe2a-a6f9-4335-b134-3a27e52c6685\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6"
Jun 15 06:45:33.382453 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.382442    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaadbe2a-a6f9-4335-b134-3a27e52c6685-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6\" (UID: \"eaadbe2a-a6f9-4335-b134-3a27e52c6685\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6"
Jun 15 06:45:33.382611 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.382471    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eaadbe2a-a6f9-4335-b134-3a27e52c6685-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6\" (UID: \"eaadbe2a-a6f9-4335-b134-3a27e52c6685\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6"
Jun 15 06:45:33.382824 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.382806    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaadbe2a-a6f9-4335-b134-3a27e52c6685-kserve-provision-location\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6\" (UID: \"eaadbe2a-a6f9-4335-b134-3a27e52c6685\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6"
Jun 15 06:45:33.383105 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.383083    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eaadbe2a-a6f9-4335-b134-3a27e52c6685-cabundle-cert\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6\" (UID: \"eaadbe2a-a6f9-4335-b134-3a27e52c6685\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6"
Jun 15 06:45:33.383152 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.383084    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaadbe2a-a6f9-4335-b134-3a27e52c6685-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6\" (UID: \"eaadbe2a-a6f9-4335-b134-3a27e52c6685\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6"
Jun 15 06:45:33.384774 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.384745    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaadbe2a-a6f9-4335-b134-3a27e52c6685-proxy-tls\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6\" (UID: \"eaadbe2a-a6f9-4335-b134-3a27e52c6685\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6"
Jun 15 06:45:33.392534 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.392504    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t2px\" (UniqueName: \"kubernetes.io/projected/eaadbe2a-a6f9-4335-b134-3a27e52c6685-kube-api-access-8t2px\") pod \"isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6\" (UID: \"eaadbe2a-a6f9-4335-b134-3a27e52c6685\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6"
Jun 15 06:45:33.475863 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.475777    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6"
Jun 15 06:45:33.608410 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:33.608387    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6"]
Jun 15 06:45:33.610658 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:45:33.610631    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaadbe2a_a6f9_4335_b134_3a27e52c6685.slice/crio-fcc2a539e03f1541a41ef7d2a5d269c7e435ca8bb8de41dd11444510f1803ea8 WatchSource:0}: Error finding container fcc2a539e03f1541a41ef7d2a5d269c7e435ca8bb8de41dd11444510f1803ea8: Status 404 returned error can't find the container with id fcc2a539e03f1541a41ef7d2a5d269c7e435ca8bb8de41dd11444510f1803ea8
Jun 15 06:45:34.607570 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:34.607538    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6" event={"ID":"eaadbe2a-a6f9-4335-b134-3a27e52c6685","Type":"ContainerStarted","Data":"d2b48f6015be4ab16327294e88d7e4d5cf467efe8e250de2cf1cee46b10aa568"}
Jun 15 06:45:34.607948 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:34.607598    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6" event={"ID":"eaadbe2a-a6f9-4335-b134-3a27e52c6685","Type":"ContainerStarted","Data":"fcc2a539e03f1541a41ef7d2a5d269c7e435ca8bb8de41dd11444510f1803ea8"}
Jun 15 06:45:38.632104 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:38.632065    2572 generic.go:356] "Generic (PLEG): container finished" podID="eaadbe2a-a6f9-4335-b134-3a27e52c6685" containerID="d2b48f6015be4ab16327294e88d7e4d5cf467efe8e250de2cf1cee46b10aa568" exitCode=0
Jun 15 06:45:38.632522 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:38.632135    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6" event={"ID":"eaadbe2a-a6f9-4335-b134-3a27e52c6685","Type":"ContainerDied","Data":"d2b48f6015be4ab16327294e88d7e4d5cf467efe8e250de2cf1cee46b10aa568"}
Jun 15 06:45:39.640529 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:39.640495    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6" event={"ID":"eaadbe2a-a6f9-4335-b134-3a27e52c6685","Type":"ContainerStarted","Data":"a2db9e328a1b861040c5d1c2238bc1c635fcdc58c037f128a48774c1bda297bb"}
Jun 15 06:45:39.641012 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:39.640560    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6" event={"ID":"eaadbe2a-a6f9-4335-b134-3a27e52c6685","Type":"ContainerStarted","Data":"088b1fa360c8057b4ac3918f16f458c998463a9ef3765276f02a40ba92a3b90b"}
Jun 15 06:45:39.641012 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:39.640765    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6"
Jun 15 06:45:39.663091 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:39.663043    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6" podStartSLOduration=2.577467683 podStartE2EDuration="6.663027678s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.085559995s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:45:39.661856884 +0000 UTC m=+3454.810036881" watchObservedRunningTime="2026-06-15 06:45:39.663027678 +0000 UTC m=+3454.811207676"
Jun 15 06:45:40.645179 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:40.645150    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6"
Jun 15 06:45:46.655813 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:45:46.655784    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6"
Jun 15 06:46:16.659875 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:16.659800    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6"
Jun 15 06:46:23.224650 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:23.224614    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6"]
Jun 15 06:46:23.225080 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:23.224832    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6" podUID="eaadbe2a-a6f9-4335-b134-3a27e52c6685" containerName="kserve-container" containerID="cri-o://088b1fa360c8057b4ac3918f16f458c998463a9ef3765276f02a40ba92a3b90b" gracePeriod=30
Jun 15 06:46:23.225080 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:23.224887    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6" podUID="eaadbe2a-a6f9-4335-b134-3a27e52c6685" containerName="kube-rbac-proxy" containerID="cri-o://a2db9e328a1b861040c5d1c2238bc1c635fcdc58c037f128a48774c1bda297bb" gracePeriod=30
Jun 15 06:46:23.291457 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:23.291421    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9"]
Jun 15 06:46:23.295479 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:23.295454    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9"
Jun 15 06:46:23.297802 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:23.297783    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-predictor-serving-cert\""
Jun 15 06:46:23.297910 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:23.297785    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"xgboost-v2-mlserver-kube-rbac-proxy-sar-config\""
Jun 15 06:46:23.308397 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:23.308353    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9"]
Jun 15 06:46:23.432730 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:23.432698    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01d2b86f-80eb-4061-93f6-6237737958c3-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9\" (UID: \"01d2b86f-80eb-4061-93f6-6237737958c3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9"
Jun 15 06:46:23.432879 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:23.432751    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01d2b86f-80eb-4061-93f6-6237737958c3-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9\" (UID: \"01d2b86f-80eb-4061-93f6-6237737958c3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9"
Jun 15 06:46:23.432879 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:23.432769    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7d67\" (UniqueName: \"kubernetes.io/projected/01d2b86f-80eb-4061-93f6-6237737958c3-kube-api-access-q7d67\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9\" (UID: \"01d2b86f-80eb-4061-93f6-6237737958c3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9"
Jun 15 06:46:23.432879 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:23.432862    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01d2b86f-80eb-4061-93f6-6237737958c3-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9\" (UID: \"01d2b86f-80eb-4061-93f6-6237737958c3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9"
Jun 15 06:46:23.433030 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:23.432893    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01d2b86f-80eb-4061-93f6-6237737958c3-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9\" (UID: \"01d2b86f-80eb-4061-93f6-6237737958c3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9"
Jun 15 06:46:23.534115 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:23.534028    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01d2b86f-80eb-4061-93f6-6237737958c3-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9\" (UID: \"01d2b86f-80eb-4061-93f6-6237737958c3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9"
Jun 15 06:46:23.534115 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:23.534090    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01d2b86f-80eb-4061-93f6-6237737958c3-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9\" (UID: \"01d2b86f-80eb-4061-93f6-6237737958c3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9"
Jun 15 06:46:23.534115 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:23.534107    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7d67\" (UniqueName: \"kubernetes.io/projected/01d2b86f-80eb-4061-93f6-6237737958c3-kube-api-access-q7d67\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9\" (UID: \"01d2b86f-80eb-4061-93f6-6237737958c3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9"
Jun 15 06:46:23.534393 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:23.534133    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01d2b86f-80eb-4061-93f6-6237737958c3-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9\" (UID: \"01d2b86f-80eb-4061-93f6-6237737958c3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9"
Jun 15 06:46:23.534393 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:23.534157    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01d2b86f-80eb-4061-93f6-6237737958c3-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9\" (UID: \"01d2b86f-80eb-4061-93f6-6237737958c3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9"
Jun 15 06:46:23.534543 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:23.534519    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01d2b86f-80eb-4061-93f6-6237737958c3-kserve-provision-location\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9\" (UID: \"01d2b86f-80eb-4061-93f6-6237737958c3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9"
Jun 15 06:46:23.534965 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:23.534943    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01d2b86f-80eb-4061-93f6-6237737958c3-cabundle-cert\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9\" (UID: \"01d2b86f-80eb-4061-93f6-6237737958c3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9"
Jun 15 06:46:23.535008 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:23.534949    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01d2b86f-80eb-4061-93f6-6237737958c3-kube-rbac-proxy-sar-config\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9\" (UID: \"01d2b86f-80eb-4061-93f6-6237737958c3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9"
Jun 15 06:46:23.536577 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:23.536560    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01d2b86f-80eb-4061-93f6-6237737958c3-proxy-tls\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9\" (UID: \"01d2b86f-80eb-4061-93f6-6237737958c3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9"
Jun 15 06:46:23.549517 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:23.549489    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7d67\" (UniqueName: \"kubernetes.io/projected/01d2b86f-80eb-4061-93f6-6237737958c3-kube-api-access-q7d67\") pod \"xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9\" (UID: \"01d2b86f-80eb-4061-93f6-6237737958c3\") " pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9"
Jun 15 06:46:23.605252 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:23.605218    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9"
Jun 15 06:46:23.738108 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:23.738080    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9"]
Jun 15 06:46:23.740034 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:46:23.740005    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d2b86f_80eb_4061_93f6_6237737958c3.slice/crio-f1620837b8533b67fb37cb4a59a148529667db1d5d326abc580e3eca186f5bd3 WatchSource:0}: Error finding container f1620837b8533b67fb37cb4a59a148529667db1d5d326abc580e3eca186f5bd3: Status 404 returned error can't find the container with id f1620837b8533b67fb37cb4a59a148529667db1d5d326abc580e3eca186f5bd3
Jun 15 06:46:23.852620 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:23.852579    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9" event={"ID":"01d2b86f-80eb-4061-93f6-6237737958c3","Type":"ContainerStarted","Data":"3f1ec9485b487a7f4c726338d6819ddfd1109e701a9d9f54ec276313f1ed8c89"}
Jun 15 06:46:23.852793 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:23.852659    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9" event={"ID":"01d2b86f-80eb-4061-93f6-6237737958c3","Type":"ContainerStarted","Data":"f1620837b8533b67fb37cb4a59a148529667db1d5d326abc580e3eca186f5bd3"}
Jun 15 06:46:23.854710 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:23.854683    2572 generic.go:356] "Generic (PLEG): container finished" podID="eaadbe2a-a6f9-4335-b134-3a27e52c6685" containerID="a2db9e328a1b861040c5d1c2238bc1c635fcdc58c037f128a48774c1bda297bb" exitCode=2
Jun 15 06:46:23.854836 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:23.854718    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6" event={"ID":"eaadbe2a-a6f9-4335-b134-3a27e52c6685","Type":"ContainerDied","Data":"a2db9e328a1b861040c5d1c2238bc1c635fcdc58c037f128a48774c1bda297bb"}
Jun 15 06:46:26.650832 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:26.650795    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6" podUID="eaadbe2a-a6f9-4335-b134-3a27e52c6685" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.56:8643/healthz\": dial tcp 10.134.0.56:8643: connect: connection refused"
Jun 15 06:46:26.657171 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:26.657144    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6" podUID="eaadbe2a-a6f9-4335-b134-3a27e52c6685" containerName="kserve-container" probeResult="failure" output="Get \"http://10.134.0.56:8080/v2/models/isvc-xgboost-v2-mlserver/ready\": dial tcp 10.134.0.56:8080: connect: connection refused"
Jun 15 06:46:27.874521 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:27.874427    2572 generic.go:356] "Generic (PLEG): container finished" podID="01d2b86f-80eb-4061-93f6-6237737958c3" containerID="3f1ec9485b487a7f4c726338d6819ddfd1109e701a9d9f54ec276313f1ed8c89" exitCode=0
Jun 15 06:46:27.874521 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:27.874504    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9" event={"ID":"01d2b86f-80eb-4061-93f6-6237737958c3","Type":"ContainerDied","Data":"3f1ec9485b487a7f4c726338d6819ddfd1109e701a9d9f54ec276313f1ed8c89"}
Jun 15 06:46:28.881027 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:28.880988    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9" event={"ID":"01d2b86f-80eb-4061-93f6-6237737958c3","Type":"ContainerStarted","Data":"9fc23812fa632023e4ee81908cd73091b84a9ced14a87c521d37e1dc52670bce"}
Jun 15 06:46:28.881532 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:28.881104    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9" event={"ID":"01d2b86f-80eb-4061-93f6-6237737958c3","Type":"ContainerStarted","Data":"543c23e6dd64b4a4957dffef0401f566689c904aa074cace115799f212cf84c6"}
Jun 15 06:46:28.881532 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:28.881337    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9"
Jun 15 06:46:28.902427 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:28.902337    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9" podStartSLOduration=2.257653323 podStartE2EDuration="5.902322423s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.6446691s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:46:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:46:28.901581814 +0000 UTC m=+3504.049761813" watchObservedRunningTime="2026-06-15 06:46:28.902322423 +0000 UTC m=+3504.050502420"
Jun 15 06:46:29.768651 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.768627    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6"
Jun 15 06:46:29.888229 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.888133    2572 generic.go:356] "Generic (PLEG): container finished" podID="eaadbe2a-a6f9-4335-b134-3a27e52c6685" containerID="088b1fa360c8057b4ac3918f16f458c998463a9ef3765276f02a40ba92a3b90b" exitCode=0
Jun 15 06:46:29.888229 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.888186    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6"
Jun 15 06:46:29.888229 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.888187    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6" event={"ID":"eaadbe2a-a6f9-4335-b134-3a27e52c6685","Type":"ContainerDied","Data":"088b1fa360c8057b4ac3918f16f458c998463a9ef3765276f02a40ba92a3b90b"}
Jun 15 06:46:29.888767 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.888248    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6" event={"ID":"eaadbe2a-a6f9-4335-b134-3a27e52c6685","Type":"ContainerDied","Data":"fcc2a539e03f1541a41ef7d2a5d269c7e435ca8bb8de41dd11444510f1803ea8"}
Jun 15 06:46:29.888767 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.888278    2572 scope.go:117] "RemoveContainer" containerID="a2db9e328a1b861040c5d1c2238bc1c635fcdc58c037f128a48774c1bda297bb"
Jun 15 06:46:29.888767 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.888702    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9"
Jun 15 06:46:29.896240 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.896218    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaadbe2a-a6f9-4335-b134-3a27e52c6685-kube-rbac-proxy-sar-config\") pod \"eaadbe2a-a6f9-4335-b134-3a27e52c6685\" (UID: \"eaadbe2a-a6f9-4335-b134-3a27e52c6685\") "
Jun 15 06:46:29.896401 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.896260    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8t2px\" (UniqueName: \"kubernetes.io/projected/eaadbe2a-a6f9-4335-b134-3a27e52c6685-kube-api-access-8t2px\") pod \"eaadbe2a-a6f9-4335-b134-3a27e52c6685\" (UID: \"eaadbe2a-a6f9-4335-b134-3a27e52c6685\") "
Jun 15 06:46:29.896401 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.896278    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaadbe2a-a6f9-4335-b134-3a27e52c6685-proxy-tls\") pod \"eaadbe2a-a6f9-4335-b134-3a27e52c6685\" (UID: \"eaadbe2a-a6f9-4335-b134-3a27e52c6685\") "
Jun 15 06:46:29.896401 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.896309    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaadbe2a-a6f9-4335-b134-3a27e52c6685-kserve-provision-location\") pod \"eaadbe2a-a6f9-4335-b134-3a27e52c6685\" (UID: \"eaadbe2a-a6f9-4335-b134-3a27e52c6685\") "
Jun 15 06:46:29.896591 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.896536    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eaadbe2a-a6f9-4335-b134-3a27e52c6685-cabundle-cert\") pod \"eaadbe2a-a6f9-4335-b134-3a27e52c6685\" (UID: \"eaadbe2a-a6f9-4335-b134-3a27e52c6685\") "
Jun 15 06:46:29.896683 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.896659    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaadbe2a-a6f9-4335-b134-3a27e52c6685-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "eaadbe2a-a6f9-4335-b134-3a27e52c6685" (UID: "eaadbe2a-a6f9-4335-b134-3a27e52c6685"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:46:29.896856 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.896658    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaadbe2a-a6f9-4335-b134-3a27e52c6685-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "eaadbe2a-a6f9-4335-b134-3a27e52c6685" (UID: "eaadbe2a-a6f9-4335-b134-3a27e52c6685"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:46:29.896929 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.896857    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaadbe2a-a6f9-4335-b134-3a27e52c6685-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "eaadbe2a-a6f9-4335-b134-3a27e52c6685" (UID: "eaadbe2a-a6f9-4335-b134-3a27e52c6685"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:46:29.896929 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.896866    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/eaadbe2a-a6f9-4335-b134-3a27e52c6685-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:46:29.896929 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.896891    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/eaadbe2a-a6f9-4335-b134-3a27e52c6685-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:46:29.897085 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.897060    2572 scope.go:117] "RemoveContainer" containerID="088b1fa360c8057b4ac3918f16f458c998463a9ef3765276f02a40ba92a3b90b"
Jun 15 06:46:29.898486 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.898463    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaadbe2a-a6f9-4335-b134-3a27e52c6685-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "eaadbe2a-a6f9-4335-b134-3a27e52c6685" (UID: "eaadbe2a-a6f9-4335-b134-3a27e52c6685"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:46:29.898595 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.898532    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaadbe2a-a6f9-4335-b134-3a27e52c6685-kube-api-access-8t2px" (OuterVolumeSpecName: "kube-api-access-8t2px") pod "eaadbe2a-a6f9-4335-b134-3a27e52c6685" (UID: "eaadbe2a-a6f9-4335-b134-3a27e52c6685"). InnerVolumeSpecName "kube-api-access-8t2px". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:46:29.913857 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.913838    2572 scope.go:117] "RemoveContainer" containerID="d2b48f6015be4ab16327294e88d7e4d5cf467efe8e250de2cf1cee46b10aa568"
Jun 15 06:46:29.921187 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.921168    2572 scope.go:117] "RemoveContainer" containerID="a2db9e328a1b861040c5d1c2238bc1c635fcdc58c037f128a48774c1bda297bb"
Jun 15 06:46:29.921522 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:46:29.921502    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2db9e328a1b861040c5d1c2238bc1c635fcdc58c037f128a48774c1bda297bb\": container with ID starting with a2db9e328a1b861040c5d1c2238bc1c635fcdc58c037f128a48774c1bda297bb not found: ID does not exist" containerID="a2db9e328a1b861040c5d1c2238bc1c635fcdc58c037f128a48774c1bda297bb"
Jun 15 06:46:29.921589 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.921529    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2db9e328a1b861040c5d1c2238bc1c635fcdc58c037f128a48774c1bda297bb"} err="failed to get container status \"a2db9e328a1b861040c5d1c2238bc1c635fcdc58c037f128a48774c1bda297bb\": rpc error: code = NotFound desc = could not find container \"a2db9e328a1b861040c5d1c2238bc1c635fcdc58c037f128a48774c1bda297bb\": container with ID starting with a2db9e328a1b861040c5d1c2238bc1c635fcdc58c037f128a48774c1bda297bb not found: ID does not exist"
Jun 15 06:46:29.921589 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.921549    2572 scope.go:117] "RemoveContainer" containerID="088b1fa360c8057b4ac3918f16f458c998463a9ef3765276f02a40ba92a3b90b"
Jun 15 06:46:29.921795 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:46:29.921778    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"088b1fa360c8057b4ac3918f16f458c998463a9ef3765276f02a40ba92a3b90b\": container with ID starting with 088b1fa360c8057b4ac3918f16f458c998463a9ef3765276f02a40ba92a3b90b not found: ID does not exist" containerID="088b1fa360c8057b4ac3918f16f458c998463a9ef3765276f02a40ba92a3b90b"
Jun 15 06:46:29.921829 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.921801    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"088b1fa360c8057b4ac3918f16f458c998463a9ef3765276f02a40ba92a3b90b"} err="failed to get container status \"088b1fa360c8057b4ac3918f16f458c998463a9ef3765276f02a40ba92a3b90b\": rpc error: code = NotFound desc = could not find container \"088b1fa360c8057b4ac3918f16f458c998463a9ef3765276f02a40ba92a3b90b\": container with ID starting with 088b1fa360c8057b4ac3918f16f458c998463a9ef3765276f02a40ba92a3b90b not found: ID does not exist"
Jun 15 06:46:29.921829 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.921819    2572 scope.go:117] "RemoveContainer" containerID="d2b48f6015be4ab16327294e88d7e4d5cf467efe8e250de2cf1cee46b10aa568"
Jun 15 06:46:29.922056 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:46:29.922036    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2b48f6015be4ab16327294e88d7e4d5cf467efe8e250de2cf1cee46b10aa568\": container with ID starting with d2b48f6015be4ab16327294e88d7e4d5cf467efe8e250de2cf1cee46b10aa568 not found: ID does not exist" containerID="d2b48f6015be4ab16327294e88d7e4d5cf467efe8e250de2cf1cee46b10aa568"
Jun 15 06:46:29.922108 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.922061    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2b48f6015be4ab16327294e88d7e4d5cf467efe8e250de2cf1cee46b10aa568"} err="failed to get container status \"d2b48f6015be4ab16327294e88d7e4d5cf467efe8e250de2cf1cee46b10aa568\": rpc error: code = NotFound desc = could not find container \"d2b48f6015be4ab16327294e88d7e4d5cf467efe8e250de2cf1cee46b10aa568\": container with ID starting with d2b48f6015be4ab16327294e88d7e4d5cf467efe8e250de2cf1cee46b10aa568 not found: ID does not exist"
Jun 15 06:46:29.998320 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.998279    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t2px\" (UniqueName: \"kubernetes.io/projected/eaadbe2a-a6f9-4335-b134-3a27e52c6685-kube-api-access-8t2px\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:46:29.998320 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.998317    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/eaadbe2a-a6f9-4335-b134-3a27e52c6685-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:46:29.998593 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:29.998336    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/eaadbe2a-a6f9-4335-b134-3a27e52c6685-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:46:30.211945 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:30.211913    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6"]
Jun 15 06:46:30.219086 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:30.219057    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-mlserver-predictor-7b55b84f54-frhg6"]
Jun 15 06:46:31.324872 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:31.324841    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaadbe2a-a6f9-4335-b134-3a27e52c6685" path="/var/lib/kubelet/pods/eaadbe2a-a6f9-4335-b134-3a27e52c6685/volumes"
Jun 15 06:46:35.900583 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:46:35.900558    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9"
Jun 15 06:47:05.905028 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:05.904995    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9"
Jun 15 06:47:13.430955 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:13.430915    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9"]
Jun 15 06:47:13.431510 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:13.431156    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9" podUID="01d2b86f-80eb-4061-93f6-6237737958c3" containerName="kserve-container" containerID="cri-o://543c23e6dd64b4a4957dffef0401f566689c904aa074cace115799f212cf84c6" gracePeriod=30
Jun 15 06:47:13.431510 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:13.431214    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9" podUID="01d2b86f-80eb-4061-93f6-6237737958c3" containerName="kube-rbac-proxy" containerID="cri-o://9fc23812fa632023e4ee81908cd73091b84a9ced14a87c521d37e1dc52670bce" gracePeriod=30
Jun 15 06:47:14.096009 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:14.095975    2572 generic.go:356] "Generic (PLEG): container finished" podID="01d2b86f-80eb-4061-93f6-6237737958c3" containerID="9fc23812fa632023e4ee81908cd73091b84a9ced14a87c521d37e1dc52670bce" exitCode=2
Jun 15 06:47:14.096229 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:14.096043    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9" event={"ID":"01d2b86f-80eb-4061-93f6-6237737958c3","Type":"ContainerDied","Data":"9fc23812fa632023e4ee81908cd73091b84a9ced14a87c521d37e1dc52670bce"}
Jun 15 06:47:15.895783 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:15.895728    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9" podUID="01d2b86f-80eb-4061-93f6-6237737958c3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
Jun 15 06:47:19.877661 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:19.877638    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9"
Jun 15 06:47:20.034824 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.034733    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01d2b86f-80eb-4061-93f6-6237737958c3-cabundle-cert\") pod \"01d2b86f-80eb-4061-93f6-6237737958c3\" (UID: \"01d2b86f-80eb-4061-93f6-6237737958c3\") "
Jun 15 06:47:20.034824 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.034815    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7d67\" (UniqueName: \"kubernetes.io/projected/01d2b86f-80eb-4061-93f6-6237737958c3-kube-api-access-q7d67\") pod \"01d2b86f-80eb-4061-93f6-6237737958c3\" (UID: \"01d2b86f-80eb-4061-93f6-6237737958c3\") "
Jun 15 06:47:20.035069 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.034841    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01d2b86f-80eb-4061-93f6-6237737958c3-proxy-tls\") pod \"01d2b86f-80eb-4061-93f6-6237737958c3\" (UID: \"01d2b86f-80eb-4061-93f6-6237737958c3\") "
Jun 15 06:47:20.035069 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.034869    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01d2b86f-80eb-4061-93f6-6237737958c3-kserve-provision-location\") pod \"01d2b86f-80eb-4061-93f6-6237737958c3\" (UID: \"01d2b86f-80eb-4061-93f6-6237737958c3\") "
Jun 15 06:47:20.035069 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.034894    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01d2b86f-80eb-4061-93f6-6237737958c3-kube-rbac-proxy-sar-config\") pod \"01d2b86f-80eb-4061-93f6-6237737958c3\" (UID: \"01d2b86f-80eb-4061-93f6-6237737958c3\") "
Jun 15 06:47:20.035256 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.035219    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01d2b86f-80eb-4061-93f6-6237737958c3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "01d2b86f-80eb-4061-93f6-6237737958c3" (UID: "01d2b86f-80eb-4061-93f6-6237737958c3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:47:20.035305 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.035260    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01d2b86f-80eb-4061-93f6-6237737958c3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "01d2b86f-80eb-4061-93f6-6237737958c3" (UID: "01d2b86f-80eb-4061-93f6-6237737958c3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:47:20.035425 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.035400    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01d2b86f-80eb-4061-93f6-6237737958c3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "01d2b86f-80eb-4061-93f6-6237737958c3" (UID: "01d2b86f-80eb-4061-93f6-6237737958c3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:47:20.036945 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.036926    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01d2b86f-80eb-4061-93f6-6237737958c3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "01d2b86f-80eb-4061-93f6-6237737958c3" (UID: "01d2b86f-80eb-4061-93f6-6237737958c3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:47:20.037053 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.037035    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01d2b86f-80eb-4061-93f6-6237737958c3-kube-api-access-q7d67" (OuterVolumeSpecName: "kube-api-access-q7d67") pod "01d2b86f-80eb-4061-93f6-6237737958c3" (UID: "01d2b86f-80eb-4061-93f6-6237737958c3"). InnerVolumeSpecName "kube-api-access-q7d67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:47:20.125829 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.125791    2572 generic.go:356] "Generic (PLEG): container finished" podID="01d2b86f-80eb-4061-93f6-6237737958c3" containerID="543c23e6dd64b4a4957dffef0401f566689c904aa074cace115799f212cf84c6" exitCode=0
Jun 15 06:47:20.126003 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.125850    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9"
Jun 15 06:47:20.126003 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.125879    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9" event={"ID":"01d2b86f-80eb-4061-93f6-6237737958c3","Type":"ContainerDied","Data":"543c23e6dd64b4a4957dffef0401f566689c904aa074cace115799f212cf84c6"}
Jun 15 06:47:20.126003 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.125936    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9" event={"ID":"01d2b86f-80eb-4061-93f6-6237737958c3","Type":"ContainerDied","Data":"f1620837b8533b67fb37cb4a59a148529667db1d5d326abc580e3eca186f5bd3"}
Jun 15 06:47:20.126003 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.125963    2572 scope.go:117] "RemoveContainer" containerID="9fc23812fa632023e4ee81908cd73091b84a9ced14a87c521d37e1dc52670bce"
Jun 15 06:47:20.134914 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.134836    2572 scope.go:117] "RemoveContainer" containerID="543c23e6dd64b4a4957dffef0401f566689c904aa074cace115799f212cf84c6"
Jun 15 06:47:20.136026 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.135776    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7d67\" (UniqueName: \"kubernetes.io/projected/01d2b86f-80eb-4061-93f6-6237737958c3-kube-api-access-q7d67\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:47:20.136026 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.135800    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/01d2b86f-80eb-4061-93f6-6237737958c3-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:47:20.136026 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.135818    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/01d2b86f-80eb-4061-93f6-6237737958c3-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:47:20.136026 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.135830    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/01d2b86f-80eb-4061-93f6-6237737958c3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:47:20.136026 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.135846    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/01d2b86f-80eb-4061-93f6-6237737958c3-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:47:20.144864 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.144838    2572 scope.go:117] "RemoveContainer" containerID="3f1ec9485b487a7f4c726338d6819ddfd1109e701a9d9f54ec276313f1ed8c89"
Jun 15 06:47:20.150489 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.150468    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9"]
Jun 15 06:47:20.152987 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.152963    2572 scope.go:117] "RemoveContainer" containerID="9fc23812fa632023e4ee81908cd73091b84a9ced14a87c521d37e1dc52670bce"
Jun 15 06:47:20.153245 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:47:20.153224    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fc23812fa632023e4ee81908cd73091b84a9ced14a87c521d37e1dc52670bce\": container with ID starting with 9fc23812fa632023e4ee81908cd73091b84a9ced14a87c521d37e1dc52670bce not found: ID does not exist" containerID="9fc23812fa632023e4ee81908cd73091b84a9ced14a87c521d37e1dc52670bce"
Jun 15 06:47:20.153308 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.153257    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fc23812fa632023e4ee81908cd73091b84a9ced14a87c521d37e1dc52670bce"} err="failed to get container status \"9fc23812fa632023e4ee81908cd73091b84a9ced14a87c521d37e1dc52670bce\": rpc error: code = NotFound desc = could not find container \"9fc23812fa632023e4ee81908cd73091b84a9ced14a87c521d37e1dc52670bce\": container with ID starting with 9fc23812fa632023e4ee81908cd73091b84a9ced14a87c521d37e1dc52670bce not found: ID does not exist"
Jun 15 06:47:20.153308 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.153277    2572 scope.go:117] "RemoveContainer" containerID="543c23e6dd64b4a4957dffef0401f566689c904aa074cace115799f212cf84c6"
Jun 15 06:47:20.153461 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.153445    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/xgboost-v2-mlserver-predictor-59944bcf4c-9lkr9"]
Jun 15 06:47:20.153503 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:47:20.153485    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"543c23e6dd64b4a4957dffef0401f566689c904aa074cace115799f212cf84c6\": container with ID starting with 543c23e6dd64b4a4957dffef0401f566689c904aa074cace115799f212cf84c6 not found: ID does not exist" containerID="543c23e6dd64b4a4957dffef0401f566689c904aa074cace115799f212cf84c6"
Jun 15 06:47:20.153538 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.153504    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"543c23e6dd64b4a4957dffef0401f566689c904aa074cace115799f212cf84c6"} err="failed to get container status \"543c23e6dd64b4a4957dffef0401f566689c904aa074cace115799f212cf84c6\": rpc error: code = NotFound desc = could not find container \"543c23e6dd64b4a4957dffef0401f566689c904aa074cace115799f212cf84c6\": container with ID starting with 543c23e6dd64b4a4957dffef0401f566689c904aa074cace115799f212cf84c6 not found: ID does not exist"
Jun 15 06:47:20.153538 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.153518    2572 scope.go:117] "RemoveContainer" containerID="3f1ec9485b487a7f4c726338d6819ddfd1109e701a9d9f54ec276313f1ed8c89"
Jun 15 06:47:20.153761 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:47:20.153745    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f1ec9485b487a7f4c726338d6819ddfd1109e701a9d9f54ec276313f1ed8c89\": container with ID starting with 3f1ec9485b487a7f4c726338d6819ddfd1109e701a9d9f54ec276313f1ed8c89 not found: ID does not exist" containerID="3f1ec9485b487a7f4c726338d6819ddfd1109e701a9d9f54ec276313f1ed8c89"
Jun 15 06:47:20.153819 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:20.153768    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f1ec9485b487a7f4c726338d6819ddfd1109e701a9d9f54ec276313f1ed8c89"} err="failed to get container status \"3f1ec9485b487a7f4c726338d6819ddfd1109e701a9d9f54ec276313f1ed8c89\": rpc error: code = NotFound desc = could not find container \"3f1ec9485b487a7f4c726338d6819ddfd1109e701a9d9f54ec276313f1ed8c89\": container with ID starting with 3f1ec9485b487a7f4c726338d6819ddfd1109e701a9d9f54ec276313f1ed8c89 not found: ID does not exist"
Jun 15 06:47:21.329469 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:47:21.329438    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01d2b86f-80eb-4061-93f6-6237737958c3" path="/var/lib/kubelet/pods/01d2b86f-80eb-4061-93f6-6237737958c3/volumes"
Jun 15 06:48:05.597910 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:05.597884    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 06:48:05.627488 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:05.627460    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 06:48:33.708529 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.708492    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf"]
Jun 15 06:48:33.709028 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.709009    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaadbe2a-a6f9-4335-b134-3a27e52c6685" containerName="kserve-container"
Jun 15 06:48:33.709028 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.709022    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaadbe2a-a6f9-4335-b134-3a27e52c6685" containerName="kserve-container"
Jun 15 06:48:33.709117 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.709038    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaadbe2a-a6f9-4335-b134-3a27e52c6685" containerName="storage-initializer"
Jun 15 06:48:33.709117 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.709045    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaadbe2a-a6f9-4335-b134-3a27e52c6685" containerName="storage-initializer"
Jun 15 06:48:33.709117 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.709054    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01d2b86f-80eb-4061-93f6-6237737958c3" containerName="kserve-container"
Jun 15 06:48:33.709117 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.709063    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="01d2b86f-80eb-4061-93f6-6237737958c3" containerName="kserve-container"
Jun 15 06:48:33.709117 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.709072    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01d2b86f-80eb-4061-93f6-6237737958c3" containerName="kube-rbac-proxy"
Jun 15 06:48:33.709117 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.709081    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="01d2b86f-80eb-4061-93f6-6237737958c3" containerName="kube-rbac-proxy"
Jun 15 06:48:33.709117 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.709091    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01d2b86f-80eb-4061-93f6-6237737958c3" containerName="storage-initializer"
Jun 15 06:48:33.709117 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.709100    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="01d2b86f-80eb-4061-93f6-6237737958c3" containerName="storage-initializer"
Jun 15 06:48:33.709407 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.709132    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaadbe2a-a6f9-4335-b134-3a27e52c6685" containerName="kube-rbac-proxy"
Jun 15 06:48:33.709407 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.709139    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaadbe2a-a6f9-4335-b134-3a27e52c6685" containerName="kube-rbac-proxy"
Jun 15 06:48:33.709407 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.709209    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaadbe2a-a6f9-4335-b134-3a27e52c6685" containerName="kserve-container"
Jun 15 06:48:33.709407 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.709216    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="01d2b86f-80eb-4061-93f6-6237737958c3" containerName="kube-rbac-proxy"
Jun 15 06:48:33.709407 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.709221    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="01d2b86f-80eb-4061-93f6-6237737958c3" containerName="kserve-container"
Jun 15 06:48:33.709407 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.709227    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaadbe2a-a6f9-4335-b134-3a27e52c6685" containerName="kube-rbac-proxy"
Jun 15 06:48:33.712584 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.712564    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf"
Jun 15 06:48:33.715546 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.715524    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jun 15 06:48:33.715742 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.715687    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 15 06:48:33.715871 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.715801    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 15 06:48:33.715871 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.715815    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jun 15 06:48:33.715871 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.715860    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-predictor-serving-cert\""
Jun 15 06:48:33.716046 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.715931    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-runtime-kube-rbac-proxy-sar-config\""
Jun 15 06:48:33.716117 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.716078    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-q7vt7\""
Jun 15 06:48:33.730824 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.730528    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf"]
Jun 15 06:48:33.774164 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.774132    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c11660ef-e630-42b9-94f4-382175a61702-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf\" (UID: \"c11660ef-e630-42b9-94f4-382175a61702\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf"
Jun 15 06:48:33.774316 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.774172    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9485z\" (UniqueName: \"kubernetes.io/projected/c11660ef-e630-42b9-94f4-382175a61702-kube-api-access-9485z\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf\" (UID: \"c11660ef-e630-42b9-94f4-382175a61702\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf"
Jun 15 06:48:33.774316 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.774244    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c11660ef-e630-42b9-94f4-382175a61702-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf\" (UID: \"c11660ef-e630-42b9-94f4-382175a61702\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf"
Jun 15 06:48:33.774316 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.774285    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c11660ef-e630-42b9-94f4-382175a61702-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf\" (UID: \"c11660ef-e630-42b9-94f4-382175a61702\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf"
Jun 15 06:48:33.774478 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.774321    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c11660ef-e630-42b9-94f4-382175a61702-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf\" (UID: \"c11660ef-e630-42b9-94f4-382175a61702\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf"
Jun 15 06:48:33.875031 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.874989    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c11660ef-e630-42b9-94f4-382175a61702-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf\" (UID: \"c11660ef-e630-42b9-94f4-382175a61702\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf"
Jun 15 06:48:33.875217 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.875036    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c11660ef-e630-42b9-94f4-382175a61702-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf\" (UID: \"c11660ef-e630-42b9-94f4-382175a61702\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf"
Jun 15 06:48:33.875217 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.875072    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c11660ef-e630-42b9-94f4-382175a61702-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf\" (UID: \"c11660ef-e630-42b9-94f4-382175a61702\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf"
Jun 15 06:48:33.875217 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.875152    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c11660ef-e630-42b9-94f4-382175a61702-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf\" (UID: \"c11660ef-e630-42b9-94f4-382175a61702\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf"
Jun 15 06:48:33.875217 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.875188    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9485z\" (UniqueName: \"kubernetes.io/projected/c11660ef-e630-42b9-94f4-382175a61702-kube-api-access-9485z\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf\" (UID: \"c11660ef-e630-42b9-94f4-382175a61702\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf"
Jun 15 06:48:33.875541 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.875518    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c11660ef-e630-42b9-94f4-382175a61702-kserve-provision-location\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf\" (UID: \"c11660ef-e630-42b9-94f4-382175a61702\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf"
Jun 15 06:48:33.875839 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.875821    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c11660ef-e630-42b9-94f4-382175a61702-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf\" (UID: \"c11660ef-e630-42b9-94f4-382175a61702\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf"
Jun 15 06:48:33.875883 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.875841    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c11660ef-e630-42b9-94f4-382175a61702-cabundle-cert\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf\" (UID: \"c11660ef-e630-42b9-94f4-382175a61702\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf"
Jun 15 06:48:33.877602 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.877581    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c11660ef-e630-42b9-94f4-382175a61702-proxy-tls\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf\" (UID: \"c11660ef-e630-42b9-94f4-382175a61702\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf"
Jun 15 06:48:33.885633 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:33.885612    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9485z\" (UniqueName: \"kubernetes.io/projected/c11660ef-e630-42b9-94f4-382175a61702-kube-api-access-9485z\") pod \"isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf\" (UID: \"c11660ef-e630-42b9-94f4-382175a61702\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf"
Jun 15 06:48:34.026842 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:34.026757    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf"
Jun 15 06:48:34.158934 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:34.158870    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf"]
Jun 15 06:48:34.461839 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:34.461804    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf" event={"ID":"c11660ef-e630-42b9-94f4-382175a61702","Type":"ContainerStarted","Data":"446b09b883c898e38f30a75f48c8f2ffb054825fa248422888579d31fc14dad4"}
Jun 15 06:48:34.462060 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:34.461861    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf" event={"ID":"c11660ef-e630-42b9-94f4-382175a61702","Type":"ContainerStarted","Data":"67f9d3ca0124db42ace465639528266fc19d05651757473a71d1f2f9f297813e"}
Jun 15 06:48:38.481881 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:38.481846    2572 generic.go:356] "Generic (PLEG): container finished" podID="c11660ef-e630-42b9-94f4-382175a61702" containerID="446b09b883c898e38f30a75f48c8f2ffb054825fa248422888579d31fc14dad4" exitCode=0
Jun 15 06:48:38.482258 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:38.481923    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf" event={"ID":"c11660ef-e630-42b9-94f4-382175a61702","Type":"ContainerDied","Data":"446b09b883c898e38f30a75f48c8f2ffb054825fa248422888579d31fc14dad4"}
Jun 15 06:48:39.488383 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:39.488329    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf" event={"ID":"c11660ef-e630-42b9-94f4-382175a61702","Type":"ContainerStarted","Data":"10048c25db1f29c322c9c9b972f5df5c08f10d79467efd60bab35596ec573dd6"}
Jun 15 06:48:39.488897 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:39.488417    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf" event={"ID":"c11660ef-e630-42b9-94f4-382175a61702","Type":"ContainerStarted","Data":"31ce957b1b19b511bd14bdf3458d61bc456982eba178a4e1dcfa319452860116"}
Jun 15 06:48:39.488897 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:39.488624    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf"
Jun 15 06:48:39.509837 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:39.509782    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf" podStartSLOduration=2.690499983 podStartE2EDuration="6.509767401s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.819267418s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:48:39.508445564 +0000 UTC m=+3634.656625560" watchObservedRunningTime="2026-06-15 06:48:39.509767401 +0000 UTC m=+3634.657947398"
Jun 15 06:48:40.493404 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:40.493374    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf"
Jun 15 06:48:46.502863 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:48:46.502837    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf"
Jun 15 06:49:16.538039 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:16.537942    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf" podUID="c11660ef-e630-42b9-94f4-382175a61702" containerName="kserve-container" probeResult="failure" output="HTTP probe failed with statuscode: 400"
Jun 15 06:49:26.505998 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:26.505969    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf"
Jun 15 06:49:33.766258 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:33.766222    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf"]
Jun 15 06:49:33.766858 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:33.766537    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf" podUID="c11660ef-e630-42b9-94f4-382175a61702" containerName="kserve-container" containerID="cri-o://31ce957b1b19b511bd14bdf3458d61bc456982eba178a4e1dcfa319452860116" gracePeriod=30
Jun 15 06:49:33.766858 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:33.766751    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf" podUID="c11660ef-e630-42b9-94f4-382175a61702" containerName="kube-rbac-proxy" containerID="cri-o://10048c25db1f29c322c9c9b972f5df5c08f10d79467efd60bab35596ec573dd6" gracePeriod=30
Jun 15 06:49:34.741213 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:34.741179    2572 generic.go:356] "Generic (PLEG): container finished" podID="c11660ef-e630-42b9-94f4-382175a61702" containerID="10048c25db1f29c322c9c9b972f5df5c08f10d79467efd60bab35596ec573dd6" exitCode=2
Jun 15 06:49:34.741431 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:34.741256    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf" event={"ID":"c11660ef-e630-42b9-94f4-382175a61702","Type":"ContainerDied","Data":"10048c25db1f29c322c9c9b972f5df5c08f10d79467efd60bab35596ec573dd6"}
Jun 15 06:49:36.498722 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:36.498683    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf" podUID="c11660ef-e630-42b9-94f4-382175a61702" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.58:8643/healthz\": dial tcp 10.134.0.58:8643: connect: connection refused"
Jun 15 06:49:41.113176 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.113152    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf"
Jun 15 06:49:41.262434 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.262338    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c11660ef-e630-42b9-94f4-382175a61702-kserve-provision-location\") pod \"c11660ef-e630-42b9-94f4-382175a61702\" (UID: \"c11660ef-e630-42b9-94f4-382175a61702\") "
Jun 15 06:49:41.262434 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.262411    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c11660ef-e630-42b9-94f4-382175a61702-kube-rbac-proxy-sar-config\") pod \"c11660ef-e630-42b9-94f4-382175a61702\" (UID: \"c11660ef-e630-42b9-94f4-382175a61702\") "
Jun 15 06:49:41.262663 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.262480    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c11660ef-e630-42b9-94f4-382175a61702-cabundle-cert\") pod \"c11660ef-e630-42b9-94f4-382175a61702\" (UID: \"c11660ef-e630-42b9-94f4-382175a61702\") "
Jun 15 06:49:41.262663 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.262544    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c11660ef-e630-42b9-94f4-382175a61702-proxy-tls\") pod \"c11660ef-e630-42b9-94f4-382175a61702\" (UID: \"c11660ef-e630-42b9-94f4-382175a61702\") "
Jun 15 06:49:41.262663 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.262574    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9485z\" (UniqueName: \"kubernetes.io/projected/c11660ef-e630-42b9-94f4-382175a61702-kube-api-access-9485z\") pod \"c11660ef-e630-42b9-94f4-382175a61702\" (UID: \"c11660ef-e630-42b9-94f4-382175a61702\") "
Jun 15 06:49:41.262814 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.262654    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c11660ef-e630-42b9-94f4-382175a61702-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c11660ef-e630-42b9-94f4-382175a61702" (UID: "c11660ef-e630-42b9-94f4-382175a61702"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:49:41.262868 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.262817    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c11660ef-e630-42b9-94f4-382175a61702-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c11660ef-e630-42b9-94f4-382175a61702" (UID: "c11660ef-e630-42b9-94f4-382175a61702"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:49:41.262868 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.262863    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c11660ef-e630-42b9-94f4-382175a61702-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:49:41.262947 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.262863    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c11660ef-e630-42b9-94f4-382175a61702-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c11660ef-e630-42b9-94f4-382175a61702" (UID: "c11660ef-e630-42b9-94f4-382175a61702"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:49:41.264562 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.264538    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c11660ef-e630-42b9-94f4-382175a61702-kube-api-access-9485z" (OuterVolumeSpecName: "kube-api-access-9485z") pod "c11660ef-e630-42b9-94f4-382175a61702" (UID: "c11660ef-e630-42b9-94f4-382175a61702"). InnerVolumeSpecName "kube-api-access-9485z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:49:41.264673 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.264594    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c11660ef-e630-42b9-94f4-382175a61702-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c11660ef-e630-42b9-94f4-382175a61702" (UID: "c11660ef-e630-42b9-94f4-382175a61702"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:49:41.364117 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.364079    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c11660ef-e630-42b9-94f4-382175a61702-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:49:41.364117 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.364109    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c11660ef-e630-42b9-94f4-382175a61702-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:49:41.364117 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.364118    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9485z\" (UniqueName: \"kubernetes.io/projected/c11660ef-e630-42b9-94f4-382175a61702-kube-api-access-9485z\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:49:41.364117 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.364126    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c11660ef-e630-42b9-94f4-382175a61702-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:49:41.779285 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.779253    2572 generic.go:356] "Generic (PLEG): container finished" podID="c11660ef-e630-42b9-94f4-382175a61702" containerID="31ce957b1b19b511bd14bdf3458d61bc456982eba178a4e1dcfa319452860116" exitCode=0
Jun 15 06:49:41.779488 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.779297    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf" event={"ID":"c11660ef-e630-42b9-94f4-382175a61702","Type":"ContainerDied","Data":"31ce957b1b19b511bd14bdf3458d61bc456982eba178a4e1dcfa319452860116"}
Jun 15 06:49:41.779488 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.779306    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf"
Jun 15 06:49:41.779488 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.779348    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf" event={"ID":"c11660ef-e630-42b9-94f4-382175a61702","Type":"ContainerDied","Data":"67f9d3ca0124db42ace465639528266fc19d05651757473a71d1f2f9f297813e"}
Jun 15 06:49:41.779488 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.779392    2572 scope.go:117] "RemoveContainer" containerID="10048c25db1f29c322c9c9b972f5df5c08f10d79467efd60bab35596ec573dd6"
Jun 15 06:49:41.787669 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.787640    2572 scope.go:117] "RemoveContainer" containerID="31ce957b1b19b511bd14bdf3458d61bc456982eba178a4e1dcfa319452860116"
Jun 15 06:49:41.795175 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.795153    2572 scope.go:117] "RemoveContainer" containerID="446b09b883c898e38f30a75f48c8f2ffb054825fa248422888579d31fc14dad4"
Jun 15 06:49:41.798687 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.798666    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf"]
Jun 15 06:49:41.802719 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.802698    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-runtime-predictor-7787c55dc9-89pdf"]
Jun 15 06:49:41.803273 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.803260    2572 scope.go:117] "RemoveContainer" containerID="10048c25db1f29c322c9c9b972f5df5c08f10d79467efd60bab35596ec573dd6"
Jun 15 06:49:41.803550 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:49:41.803524    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10048c25db1f29c322c9c9b972f5df5c08f10d79467efd60bab35596ec573dd6\": container with ID starting with 10048c25db1f29c322c9c9b972f5df5c08f10d79467efd60bab35596ec573dd6 not found: ID does not exist" containerID="10048c25db1f29c322c9c9b972f5df5c08f10d79467efd60bab35596ec573dd6"
Jun 15 06:49:41.803617 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.803554    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10048c25db1f29c322c9c9b972f5df5c08f10d79467efd60bab35596ec573dd6"} err="failed to get container status \"10048c25db1f29c322c9c9b972f5df5c08f10d79467efd60bab35596ec573dd6\": rpc error: code = NotFound desc = could not find container \"10048c25db1f29c322c9c9b972f5df5c08f10d79467efd60bab35596ec573dd6\": container with ID starting with 10048c25db1f29c322c9c9b972f5df5c08f10d79467efd60bab35596ec573dd6 not found: ID does not exist"
Jun 15 06:49:41.803617 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.803573    2572 scope.go:117] "RemoveContainer" containerID="31ce957b1b19b511bd14bdf3458d61bc456982eba178a4e1dcfa319452860116"
Jun 15 06:49:41.803801 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:49:41.803771    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31ce957b1b19b511bd14bdf3458d61bc456982eba178a4e1dcfa319452860116\": container with ID starting with 31ce957b1b19b511bd14bdf3458d61bc456982eba178a4e1dcfa319452860116 not found: ID does not exist" containerID="31ce957b1b19b511bd14bdf3458d61bc456982eba178a4e1dcfa319452860116"
Jun 15 06:49:41.803876 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.803801    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31ce957b1b19b511bd14bdf3458d61bc456982eba178a4e1dcfa319452860116"} err="failed to get container status \"31ce957b1b19b511bd14bdf3458d61bc456982eba178a4e1dcfa319452860116\": rpc error: code = NotFound desc = could not find container \"31ce957b1b19b511bd14bdf3458d61bc456982eba178a4e1dcfa319452860116\": container with ID starting with 31ce957b1b19b511bd14bdf3458d61bc456982eba178a4e1dcfa319452860116 not found: ID does not exist"
Jun 15 06:49:41.803876 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.803824    2572 scope.go:117] "RemoveContainer" containerID="446b09b883c898e38f30a75f48c8f2ffb054825fa248422888579d31fc14dad4"
Jun 15 06:49:41.804045 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:49:41.804030    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"446b09b883c898e38f30a75f48c8f2ffb054825fa248422888579d31fc14dad4\": container with ID starting with 446b09b883c898e38f30a75f48c8f2ffb054825fa248422888579d31fc14dad4 not found: ID does not exist" containerID="446b09b883c898e38f30a75f48c8f2ffb054825fa248422888579d31fc14dad4"
Jun 15 06:49:41.804221 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:41.804050    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"446b09b883c898e38f30a75f48c8f2ffb054825fa248422888579d31fc14dad4"} err="failed to get container status \"446b09b883c898e38f30a75f48c8f2ffb054825fa248422888579d31fc14dad4\": rpc error: code = NotFound desc = could not find container \"446b09b883c898e38f30a75f48c8f2ffb054825fa248422888579d31fc14dad4\": container with ID starting with 446b09b883c898e38f30a75f48c8f2ffb054825fa248422888579d31fc14dad4 not found: ID does not exist"
Jun 15 06:49:43.324002 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:49:43.323971    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c11660ef-e630-42b9-94f4-382175a61702" path="/var/lib/kubelet/pods/c11660ef-e630-42b9-94f4-382175a61702/volumes"
Jun 15 06:50:44.046672 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.046629    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d"]
Jun 15 06:50:44.047197 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.047157    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c11660ef-e630-42b9-94f4-382175a61702" containerName="kserve-container"
Jun 15 06:50:44.047197 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.047177    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c11660ef-e630-42b9-94f4-382175a61702" containerName="kserve-container"
Jun 15 06:50:44.047317 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.047210    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c11660ef-e630-42b9-94f4-382175a61702" containerName="storage-initializer"
Jun 15 06:50:44.047317 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.047219    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c11660ef-e630-42b9-94f4-382175a61702" containerName="storage-initializer"
Jun 15 06:50:44.047317 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.047234    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c11660ef-e630-42b9-94f4-382175a61702" containerName="kube-rbac-proxy"
Jun 15 06:50:44.047317 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.047243    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="c11660ef-e630-42b9-94f4-382175a61702" containerName="kube-rbac-proxy"
Jun 15 06:50:44.047568 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.047466    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c11660ef-e630-42b9-94f4-382175a61702" containerName="kube-rbac-proxy"
Jun 15 06:50:44.047568 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.047489    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="c11660ef-e630-42b9-94f4-382175a61702" containerName="kserve-container"
Jun 15 06:50:44.050879 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.050857    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d"
Jun 15 06:50:44.053446 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.053424    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jun 15 06:50:44.053575 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.053424    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jun 15 06:50:44.053648 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.053639    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-q7vt7\""
Jun 15 06:50:44.053705 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.053641    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jun 15 06:50:44.054051 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.054037    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-predictor-serving-cert\""
Jun 15 06:50:44.054116 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.054101    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-kube-rbac-proxy-sar-config\""
Jun 15 06:50:44.061639 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.061621    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d"]
Jun 15 06:50:44.112452 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.112419    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg5t2\" (UniqueName: \"kubernetes.io/projected/d8620513-f166-45c8-aefe-0aa14fe5d0b6-kube-api-access-gg5t2\") pod \"isvc-sklearn-s3-predictor-5d7466784d-6m52d\" (UID: \"d8620513-f166-45c8-aefe-0aa14fe5d0b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d"
Jun 15 06:50:44.112628 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.112485    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8620513-f166-45c8-aefe-0aa14fe5d0b6-proxy-tls\") pod \"isvc-sklearn-s3-predictor-5d7466784d-6m52d\" (UID: \"d8620513-f166-45c8-aefe-0aa14fe5d0b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d"
Jun 15 06:50:44.112628 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.112518    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8620513-f166-45c8-aefe-0aa14fe5d0b6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-5d7466784d-6m52d\" (UID: \"d8620513-f166-45c8-aefe-0aa14fe5d0b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d"
Jun 15 06:50:44.112628 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.112539    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8620513-f166-45c8-aefe-0aa14fe5d0b6-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-5d7466784d-6m52d\" (UID: \"d8620513-f166-45c8-aefe-0aa14fe5d0b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d"
Jun 15 06:50:44.213065 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.213028    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg5t2\" (UniqueName: \"kubernetes.io/projected/d8620513-f166-45c8-aefe-0aa14fe5d0b6-kube-api-access-gg5t2\") pod \"isvc-sklearn-s3-predictor-5d7466784d-6m52d\" (UID: \"d8620513-f166-45c8-aefe-0aa14fe5d0b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d"
Jun 15 06:50:44.213219 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.213089    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8620513-f166-45c8-aefe-0aa14fe5d0b6-proxy-tls\") pod \"isvc-sklearn-s3-predictor-5d7466784d-6m52d\" (UID: \"d8620513-f166-45c8-aefe-0aa14fe5d0b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d"
Jun 15 06:50:44.213219 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.213119    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8620513-f166-45c8-aefe-0aa14fe5d0b6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-5d7466784d-6m52d\" (UID: \"d8620513-f166-45c8-aefe-0aa14fe5d0b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d"
Jun 15 06:50:44.213219 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.213138    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8620513-f166-45c8-aefe-0aa14fe5d0b6-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-5d7466784d-6m52d\" (UID: \"d8620513-f166-45c8-aefe-0aa14fe5d0b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d"
Jun 15 06:50:44.213407 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:50:44.213250    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-predictor-serving-cert: secret "isvc-sklearn-s3-predictor-serving-cert" not found
Jun 15 06:50:44.213407 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:50:44.213353    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8620513-f166-45c8-aefe-0aa14fe5d0b6-proxy-tls podName:d8620513-f166-45c8-aefe-0aa14fe5d0b6 nodeName:}" failed. No retries permitted until 2026-06-15 06:50:44.713311158 +0000 UTC m=+3759.861491134 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d8620513-f166-45c8-aefe-0aa14fe5d0b6-proxy-tls") pod "isvc-sklearn-s3-predictor-5d7466784d-6m52d" (UID: "d8620513-f166-45c8-aefe-0aa14fe5d0b6") : secret "isvc-sklearn-s3-predictor-serving-cert" not found
Jun 15 06:50:44.213584 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.213565    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8620513-f166-45c8-aefe-0aa14fe5d0b6-kserve-provision-location\") pod \"isvc-sklearn-s3-predictor-5d7466784d-6m52d\" (UID: \"d8620513-f166-45c8-aefe-0aa14fe5d0b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d"
Jun 15 06:50:44.213824 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.213806    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8620513-f166-45c8-aefe-0aa14fe5d0b6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-predictor-5d7466784d-6m52d\" (UID: \"d8620513-f166-45c8-aefe-0aa14fe5d0b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d"
Jun 15 06:50:44.227519 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.227490    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg5t2\" (UniqueName: \"kubernetes.io/projected/d8620513-f166-45c8-aefe-0aa14fe5d0b6-kube-api-access-gg5t2\") pod \"isvc-sklearn-s3-predictor-5d7466784d-6m52d\" (UID: \"d8620513-f166-45c8-aefe-0aa14fe5d0b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d"
Jun 15 06:50:44.717085 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.716898    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8620513-f166-45c8-aefe-0aa14fe5d0b6-proxy-tls\") pod \"isvc-sklearn-s3-predictor-5d7466784d-6m52d\" (UID: \"d8620513-f166-45c8-aefe-0aa14fe5d0b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d"
Jun 15 06:50:44.723476 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.720574    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8620513-f166-45c8-aefe-0aa14fe5d0b6-proxy-tls\") pod \"isvc-sklearn-s3-predictor-5d7466784d-6m52d\" (UID: \"d8620513-f166-45c8-aefe-0aa14fe5d0b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d"
Jun 15 06:50:44.962648 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:44.962614    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d"
Jun 15 06:50:45.103190 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:45.103166    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d"]
Jun 15 06:50:45.105536 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:50:45.105505    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8620513_f166_45c8_aefe_0aa14fe5d0b6.slice/crio-e88768ef35799817e5dd69cfe90b566fc0ef9c3211eeee9512cb4f7bcd092960 WatchSource:0}: Error finding container e88768ef35799817e5dd69cfe90b566fc0ef9c3211eeee9512cb4f7bcd092960: Status 404 returned error can't find the container with id e88768ef35799817e5dd69cfe90b566fc0ef9c3211eeee9512cb4f7bcd092960
Jun 15 06:50:46.073987 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:46.073952    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d" event={"ID":"d8620513-f166-45c8-aefe-0aa14fe5d0b6","Type":"ContainerStarted","Data":"a9d7d8b8ca51d006c6539fc628ced4ded0dd6d58e47f8b9aa2f2aaf985151fc7"}
Jun 15 06:50:46.074173 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:46.074012    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d" event={"ID":"d8620513-f166-45c8-aefe-0aa14fe5d0b6","Type":"ContainerStarted","Data":"e88768ef35799817e5dd69cfe90b566fc0ef9c3211eeee9512cb4f7bcd092960"}
Jun 15 06:50:47.080234 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:47.080193    2572 generic.go:356] "Generic (PLEG): container finished" podID="d8620513-f166-45c8-aefe-0aa14fe5d0b6" containerID="a9d7d8b8ca51d006c6539fc628ced4ded0dd6d58e47f8b9aa2f2aaf985151fc7" exitCode=0
Jun 15 06:50:47.080735 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:47.080246    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d" event={"ID":"d8620513-f166-45c8-aefe-0aa14fe5d0b6","Type":"ContainerDied","Data":"a9d7d8b8ca51d006c6539fc628ced4ded0dd6d58e47f8b9aa2f2aaf985151fc7"}
Jun 15 06:50:48.086963 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:48.086924    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d" event={"ID":"d8620513-f166-45c8-aefe-0aa14fe5d0b6","Type":"ContainerStarted","Data":"c9db77732118348296103121e20bf8d751f5cdaa10f128cc5c240a50472e6abe"}
Jun 15 06:50:48.087478 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:48.086990    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d" event={"ID":"d8620513-f166-45c8-aefe-0aa14fe5d0b6","Type":"ContainerStarted","Data":"5b3632ff509cfa633a388f73354448675310cf2bb7b9b3a88b7f8ff9c238efcd"}
Jun 15 06:50:48.087478 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:48.087049    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d"
Jun 15 06:50:48.107987 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:48.107937    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d" podStartSLOduration=3.110239521 podStartE2EDuration="4.107923284s" totalImagesPullingTime="0s" totalInitContainerRuntime="997.683763ms" isStatefulPod=false podCreationTimestamp="2026-06-15 06:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:50:48.106487615 +0000 UTC m=+3763.254667612" watchObservedRunningTime="2026-06-15 06:50:48.107923284 +0000 UTC m=+3763.256103280"
Jun 15 06:50:49.092124 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:49.092095    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d"
Jun 15 06:50:49.093420 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:49.093389    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d" podUID="d8620513-f166-45c8-aefe-0aa14fe5d0b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jun 15 06:50:50.096624 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:50.096583    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d" podUID="d8620513-f166-45c8-aefe-0aa14fe5d0b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jun 15 06:50:55.100764 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:55.100734    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d"
Jun 15 06:50:55.101196 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:50:55.101172    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d" podUID="d8620513-f166-45c8-aefe-0aa14fe5d0b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jun 15 06:51:05.101333 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:51:05.101290    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d" podUID="d8620513-f166-45c8-aefe-0aa14fe5d0b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jun 15 06:51:15.102012 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:51:15.101971    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d" podUID="d8620513-f166-45c8-aefe-0aa14fe5d0b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jun 15 06:51:25.101198 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:51:25.101159    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d" podUID="d8620513-f166-45c8-aefe-0aa14fe5d0b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jun 15 06:51:35.101554 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:51:35.101512    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d" podUID="d8620513-f166-45c8-aefe-0aa14fe5d0b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jun 15 06:51:45.101548 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:51:45.101505    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d" podUID="d8620513-f166-45c8-aefe-0aa14fe5d0b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jun 15 06:51:55.101606 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:51:55.101561    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d" podUID="d8620513-f166-45c8-aefe-0aa14fe5d0b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jun 15 06:51:57.319599 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:51:57.319554    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d" podUID="d8620513-f166-45c8-aefe-0aa14fe5d0b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jun 15 06:52:07.324116 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:07.324088    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d"
Jun 15 06:52:14.222280 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:14.222239    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d"]
Jun 15 06:52:14.222796 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:14.222487    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d" podUID="d8620513-f166-45c8-aefe-0aa14fe5d0b6" containerName="kserve-container" containerID="cri-o://5b3632ff509cfa633a388f73354448675310cf2bb7b9b3a88b7f8ff9c238efcd" gracePeriod=30
Jun 15 06:52:14.222796 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:14.222527    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d" podUID="d8620513-f166-45c8-aefe-0aa14fe5d0b6" containerName="kube-rbac-proxy" containerID="cri-o://c9db77732118348296103121e20bf8d751f5cdaa10f128cc5c240a50472e6abe" gracePeriod=30
Jun 15 06:52:14.370464 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:14.370430    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"]
Jun 15 06:52:14.374096 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:14.374072    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"
Jun 15 06:52:14.376187 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:14.376167    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Jun 15 06:52:14.376301 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:14.376187    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jun 15 06:52:14.376398 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:14.376310    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Jun 15 06:52:14.385013 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:14.384990    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"]
Jun 15 06:52:14.480078 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:14.479982    2572 generic.go:356] "Generic (PLEG): container finished" podID="d8620513-f166-45c8-aefe-0aa14fe5d0b6" containerID="c9db77732118348296103121e20bf8d751f5cdaa10f128cc5c240a50472e6abe" exitCode=2
Jun 15 06:52:14.480078 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:14.480059    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d" event={"ID":"d8620513-f166-45c8-aefe-0aa14fe5d0b6","Type":"ContainerDied","Data":"c9db77732118348296103121e20bf8d751f5cdaa10f128cc5c240a50472e6abe"}
Jun 15 06:52:14.504177 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:14.504149    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c887260-5348-43db-a90c-98c687c08f8c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz\" (UID: \"5c887260-5348-43db-a90c-98c687c08f8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"
Jun 15 06:52:14.504289 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:14.504180    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c887260-5348-43db-a90c-98c687c08f8c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz\" (UID: \"5c887260-5348-43db-a90c-98c687c08f8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"
Jun 15 06:52:14.504289 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:14.504203    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c887260-5348-43db-a90c-98c687c08f8c-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz\" (UID: \"5c887260-5348-43db-a90c-98c687c08f8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"
Jun 15 06:52:14.504289 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:14.504250    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c887260-5348-43db-a90c-98c687c08f8c-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz\" (UID: \"5c887260-5348-43db-a90c-98c687c08f8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"
Jun 15 06:52:14.504436 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:14.504298    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7gzw\" (UniqueName: \"kubernetes.io/projected/5c887260-5348-43db-a90c-98c687c08f8c-kube-api-access-t7gzw\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz\" (UID: \"5c887260-5348-43db-a90c-98c687c08f8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"
Jun 15 06:52:14.605210 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:14.605163    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c887260-5348-43db-a90c-98c687c08f8c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz\" (UID: \"5c887260-5348-43db-a90c-98c687c08f8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"
Jun 15 06:52:14.605210 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:14.605208    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c887260-5348-43db-a90c-98c687c08f8c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz\" (UID: \"5c887260-5348-43db-a90c-98c687c08f8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"
Jun 15 06:52:14.605515 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:14.605235    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c887260-5348-43db-a90c-98c687c08f8c-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz\" (UID: \"5c887260-5348-43db-a90c-98c687c08f8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"
Jun 15 06:52:14.605515 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:14.605269    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c887260-5348-43db-a90c-98c687c08f8c-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz\" (UID: \"5c887260-5348-43db-a90c-98c687c08f8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"
Jun 15 06:52:14.605515 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:14.605328    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7gzw\" (UniqueName: \"kubernetes.io/projected/5c887260-5348-43db-a90c-98c687c08f8c-kube-api-access-t7gzw\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz\" (UID: \"5c887260-5348-43db-a90c-98c687c08f8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"
Jun 15 06:52:14.605515 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:52:14.605476    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Jun 15 06:52:14.605676 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:52:14.605548    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5c887260-5348-43db-a90c-98c687c08f8c-proxy-tls podName:5c887260-5348-43db-a90c-98c687c08f8c nodeName:}" failed. No retries permitted until 2026-06-15 06:52:15.105528821 +0000 UTC m=+3850.253708796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5c887260-5348-43db-a90c-98c687c08f8c-proxy-tls") pod "isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz" (UID: "5c887260-5348-43db-a90c-98c687c08f8c") : secret "isvc-sklearn-s3-tls-global-pass-predictor-serving-cert" not found
Jun 15 06:52:14.605728 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:14.605671    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c887260-5348-43db-a90c-98c687c08f8c-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz\" (UID: \"5c887260-5348-43db-a90c-98c687c08f8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"
Jun 15 06:52:14.605951 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:14.605933    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c887260-5348-43db-a90c-98c687c08f8c-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz\" (UID: \"5c887260-5348-43db-a90c-98c687c08f8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"
Jun 15 06:52:14.605993 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:14.605955    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c887260-5348-43db-a90c-98c687c08f8c-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz\" (UID: \"5c887260-5348-43db-a90c-98c687c08f8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"
Jun 15 06:52:14.614875 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:14.614850    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7gzw\" (UniqueName: \"kubernetes.io/projected/5c887260-5348-43db-a90c-98c687c08f8c-kube-api-access-t7gzw\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz\" (UID: \"5c887260-5348-43db-a90c-98c687c08f8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"
Jun 15 06:52:15.097531 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:15.097485    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d" podUID="d8620513-f166-45c8-aefe-0aa14fe5d0b6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.59:8643/healthz\": dial tcp 10.134.0.59:8643: connect: connection refused"
Jun 15 06:52:15.110436 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:15.110404    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c887260-5348-43db-a90c-98c687c08f8c-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz\" (UID: \"5c887260-5348-43db-a90c-98c687c08f8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"
Jun 15 06:52:15.112836 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:15.112812    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c887260-5348-43db-a90c-98c687c08f8c-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz\" (UID: \"5c887260-5348-43db-a90c-98c687c08f8c\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"
Jun 15 06:52:15.287279 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:15.287184    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"
Jun 15 06:52:15.421087 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:15.421062    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"]
Jun 15 06:52:15.423317 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:52:15.423290    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c887260_5348_43db_a90c_98c687c08f8c.slice/crio-60f656c0f33343c7ded1e73e47a2700955bfa10523d6e5d495b4a8f2f0007df7 WatchSource:0}: Error finding container 60f656c0f33343c7ded1e73e47a2700955bfa10523d6e5d495b4a8f2f0007df7: Status 404 returned error can't find the container with id 60f656c0f33343c7ded1e73e47a2700955bfa10523d6e5d495b4a8f2f0007df7
Jun 15 06:52:15.486286 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:15.486252    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz" event={"ID":"5c887260-5348-43db-a90c-98c687c08f8c","Type":"ContainerStarted","Data":"8f1697aaed5f37122ac5f02987e3262c8fc6efdb2271030d33c782a5e735e560"}
Jun 15 06:52:15.486442 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:15.486330    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz" event={"ID":"5c887260-5348-43db-a90c-98c687c08f8c","Type":"ContainerStarted","Data":"60f656c0f33343c7ded1e73e47a2700955bfa10523d6e5d495b4a8f2f0007df7"}
Jun 15 06:52:17.320002 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:17.319961    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d" podUID="d8620513-f166-45c8-aefe-0aa14fe5d0b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.59:8080: connect: connection refused"
Jun 15 06:52:17.497349 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:17.497313    2572 generic.go:356] "Generic (PLEG): container finished" podID="5c887260-5348-43db-a90c-98c687c08f8c" containerID="8f1697aaed5f37122ac5f02987e3262c8fc6efdb2271030d33c782a5e735e560" exitCode=0
Jun 15 06:52:17.497509 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:17.497398    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz" event={"ID":"5c887260-5348-43db-a90c-98c687c08f8c","Type":"ContainerDied","Data":"8f1697aaed5f37122ac5f02987e3262c8fc6efdb2271030d33c782a5e735e560"}
Jun 15 06:52:18.504510 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:18.504472    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz" event={"ID":"5c887260-5348-43db-a90c-98c687c08f8c","Type":"ContainerStarted","Data":"c1fcb3681313b7051ebf4d0bc346e7420e5ba8a5c058512fdab30cdacae353a2"}
Jun 15 06:52:18.504950 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:18.504564    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz" event={"ID":"5c887260-5348-43db-a90c-98c687c08f8c","Type":"ContainerStarted","Data":"684e2c12f6dfe151adf685fcee78b6a07696b036caa96ec25630bff133b72978"}
Jun 15 06:52:18.504950 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:18.504868    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"
Jun 15 06:52:18.504950 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:18.504910    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"
Jun 15 06:52:18.506302 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:18.506276    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz" podUID="5c887260-5348-43db-a90c-98c687c08f8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jun 15 06:52:18.524816 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:18.524767    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz" podStartSLOduration=3.5019054880000002 podStartE2EDuration="4.5247528s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.022847312s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:52:18.52290446 +0000 UTC m=+3853.671084482" watchObservedRunningTime="2026-06-15 06:52:18.5247528 +0000 UTC m=+3853.672932798"
Jun 15 06:52:19.071504 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:19.071475    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d"
Jun 15 06:52:19.245151 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:19.245123    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gg5t2\" (UniqueName: \"kubernetes.io/projected/d8620513-f166-45c8-aefe-0aa14fe5d0b6-kube-api-access-gg5t2\") pod \"d8620513-f166-45c8-aefe-0aa14fe5d0b6\" (UID: \"d8620513-f166-45c8-aefe-0aa14fe5d0b6\") "
Jun 15 06:52:19.245311 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:19.245180    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8620513-f166-45c8-aefe-0aa14fe5d0b6-proxy-tls\") pod \"d8620513-f166-45c8-aefe-0aa14fe5d0b6\" (UID: \"d8620513-f166-45c8-aefe-0aa14fe5d0b6\") "
Jun 15 06:52:19.245311 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:19.245225    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8620513-f166-45c8-aefe-0aa14fe5d0b6-kube-rbac-proxy-sar-config\") pod \"d8620513-f166-45c8-aefe-0aa14fe5d0b6\" (UID: \"d8620513-f166-45c8-aefe-0aa14fe5d0b6\") "
Jun 15 06:52:19.245311 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:19.245254    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8620513-f166-45c8-aefe-0aa14fe5d0b6-kserve-provision-location\") pod \"d8620513-f166-45c8-aefe-0aa14fe5d0b6\" (UID: \"d8620513-f166-45c8-aefe-0aa14fe5d0b6\") "
Jun 15 06:52:19.245612 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:19.245584    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8620513-f166-45c8-aefe-0aa14fe5d0b6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d8620513-f166-45c8-aefe-0aa14fe5d0b6" (UID: "d8620513-f166-45c8-aefe-0aa14fe5d0b6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:52:19.245675 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:19.245592    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8620513-f166-45c8-aefe-0aa14fe5d0b6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d8620513-f166-45c8-aefe-0aa14fe5d0b6" (UID: "d8620513-f166-45c8-aefe-0aa14fe5d0b6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:52:19.247391 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:19.247350    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8620513-f166-45c8-aefe-0aa14fe5d0b6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d8620513-f166-45c8-aefe-0aa14fe5d0b6" (UID: "d8620513-f166-45c8-aefe-0aa14fe5d0b6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:52:19.247391 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:19.247377    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8620513-f166-45c8-aefe-0aa14fe5d0b6-kube-api-access-gg5t2" (OuterVolumeSpecName: "kube-api-access-gg5t2") pod "d8620513-f166-45c8-aefe-0aa14fe5d0b6" (UID: "d8620513-f166-45c8-aefe-0aa14fe5d0b6"). InnerVolumeSpecName "kube-api-access-gg5t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:52:19.346493 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:19.346465    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8620513-f166-45c8-aefe-0aa14fe5d0b6-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:52:19.346493 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:19.346490    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8620513-f166-45c8-aefe-0aa14fe5d0b6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:52:19.346653 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:19.346501    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d8620513-f166-45c8-aefe-0aa14fe5d0b6-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:52:19.346653 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:19.346510    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg5t2\" (UniqueName: \"kubernetes.io/projected/d8620513-f166-45c8-aefe-0aa14fe5d0b6-kube-api-access-gg5t2\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:52:19.511220 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:19.511136    2572 generic.go:356] "Generic (PLEG): container finished" podID="d8620513-f166-45c8-aefe-0aa14fe5d0b6" containerID="5b3632ff509cfa633a388f73354448675310cf2bb7b9b3a88b7f8ff9c238efcd" exitCode=0
Jun 15 06:52:19.511220 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:19.511189    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d"
Jun 15 06:52:19.511220 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:19.511212    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d" event={"ID":"d8620513-f166-45c8-aefe-0aa14fe5d0b6","Type":"ContainerDied","Data":"5b3632ff509cfa633a388f73354448675310cf2bb7b9b3a88b7f8ff9c238efcd"}
Jun 15 06:52:19.511747 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:19.511292    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d" event={"ID":"d8620513-f166-45c8-aefe-0aa14fe5d0b6","Type":"ContainerDied","Data":"e88768ef35799817e5dd69cfe90b566fc0ef9c3211eeee9512cb4f7bcd092960"}
Jun 15 06:52:19.511747 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:19.511326    2572 scope.go:117] "RemoveContainer" containerID="c9db77732118348296103121e20bf8d751f5cdaa10f128cc5c240a50472e6abe"
Jun 15 06:52:19.511747 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:19.511719    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz" podUID="5c887260-5348-43db-a90c-98c687c08f8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jun 15 06:52:19.519823 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:19.519803    2572 scope.go:117] "RemoveContainer" containerID="5b3632ff509cfa633a388f73354448675310cf2bb7b9b3a88b7f8ff9c238efcd"
Jun 15 06:52:19.527335 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:19.527283    2572 scope.go:117] "RemoveContainer" containerID="a9d7d8b8ca51d006c6539fc628ced4ded0dd6d58e47f8b9aa2f2aaf985151fc7"
Jun 15 06:52:19.530176 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:19.530149    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d"]
Jun 15 06:52:19.535588 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:19.535561    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-predictor-5d7466784d-6m52d"]
Jun 15 06:52:19.536263 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:19.536247    2572 scope.go:117] "RemoveContainer" containerID="c9db77732118348296103121e20bf8d751f5cdaa10f128cc5c240a50472e6abe"
Jun 15 06:52:19.536592 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:52:19.536573    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9db77732118348296103121e20bf8d751f5cdaa10f128cc5c240a50472e6abe\": container with ID starting with c9db77732118348296103121e20bf8d751f5cdaa10f128cc5c240a50472e6abe not found: ID does not exist" containerID="c9db77732118348296103121e20bf8d751f5cdaa10f128cc5c240a50472e6abe"
Jun 15 06:52:19.536660 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:19.536603    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9db77732118348296103121e20bf8d751f5cdaa10f128cc5c240a50472e6abe"} err="failed to get container status \"c9db77732118348296103121e20bf8d751f5cdaa10f128cc5c240a50472e6abe\": rpc error: code = NotFound desc = could not find container \"c9db77732118348296103121e20bf8d751f5cdaa10f128cc5c240a50472e6abe\": container with ID starting with c9db77732118348296103121e20bf8d751f5cdaa10f128cc5c240a50472e6abe not found: ID does not exist"
Jun 15 06:52:19.536660 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:19.536626    2572 scope.go:117] "RemoveContainer" containerID="5b3632ff509cfa633a388f73354448675310cf2bb7b9b3a88b7f8ff9c238efcd"
Jun 15 06:52:19.536850 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:52:19.536834    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b3632ff509cfa633a388f73354448675310cf2bb7b9b3a88b7f8ff9c238efcd\": container with ID starting with 5b3632ff509cfa633a388f73354448675310cf2bb7b9b3a88b7f8ff9c238efcd not found: ID does not exist" containerID="5b3632ff509cfa633a388f73354448675310cf2bb7b9b3a88b7f8ff9c238efcd"
Jun 15 06:52:19.536916 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:19.536857    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b3632ff509cfa633a388f73354448675310cf2bb7b9b3a88b7f8ff9c238efcd"} err="failed to get container status \"5b3632ff509cfa633a388f73354448675310cf2bb7b9b3a88b7f8ff9c238efcd\": rpc error: code = NotFound desc = could not find container \"5b3632ff509cfa633a388f73354448675310cf2bb7b9b3a88b7f8ff9c238efcd\": container with ID starting with 5b3632ff509cfa633a388f73354448675310cf2bb7b9b3a88b7f8ff9c238efcd not found: ID does not exist"
Jun 15 06:52:19.536916 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:19.536876    2572 scope.go:117] "RemoveContainer" containerID="a9d7d8b8ca51d006c6539fc628ced4ded0dd6d58e47f8b9aa2f2aaf985151fc7"
Jun 15 06:52:19.537134 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:52:19.537117    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9d7d8b8ca51d006c6539fc628ced4ded0dd6d58e47f8b9aa2f2aaf985151fc7\": container with ID starting with a9d7d8b8ca51d006c6539fc628ced4ded0dd6d58e47f8b9aa2f2aaf985151fc7 not found: ID does not exist" containerID="a9d7d8b8ca51d006c6539fc628ced4ded0dd6d58e47f8b9aa2f2aaf985151fc7"
Jun 15 06:52:19.537183 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:19.537140    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9d7d8b8ca51d006c6539fc628ced4ded0dd6d58e47f8b9aa2f2aaf985151fc7"} err="failed to get container status \"a9d7d8b8ca51d006c6539fc628ced4ded0dd6d58e47f8b9aa2f2aaf985151fc7\": rpc error: code = NotFound desc = could not find container \"a9d7d8b8ca51d006c6539fc628ced4ded0dd6d58e47f8b9aa2f2aaf985151fc7\": container with ID starting with a9d7d8b8ca51d006c6539fc628ced4ded0dd6d58e47f8b9aa2f2aaf985151fc7 not found: ID does not exist"
Jun 15 06:52:21.324282 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:21.324250    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8620513-f166-45c8-aefe-0aa14fe5d0b6" path="/var/lib/kubelet/pods/d8620513-f166-45c8-aefe-0aa14fe5d0b6/volumes"
Jun 15 06:52:24.516440 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:24.516411    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"
Jun 15 06:52:24.516966 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:24.516936    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz" podUID="5c887260-5348-43db-a90c-98c687c08f8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jun 15 06:52:34.517429 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:34.517385    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz" podUID="5c887260-5348-43db-a90c-98c687c08f8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jun 15 06:52:44.517661 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:44.517621    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz" podUID="5c887260-5348-43db-a90c-98c687c08f8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jun 15 06:52:54.516861 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:52:54.516823    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz" podUID="5c887260-5348-43db-a90c-98c687c08f8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jun 15 06:53:04.517910 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:04.517871    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz" podUID="5c887260-5348-43db-a90c-98c687c08f8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jun 15 06:53:05.629174 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:05.629146    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 06:53:05.663378 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:05.663343    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 06:53:14.517576 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:14.517533    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz" podUID="5c887260-5348-43db-a90c-98c687c08f8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jun 15 06:53:24.517030 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:24.516986    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz" podUID="5c887260-5348-43db-a90c-98c687c08f8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jun 15 06:53:27.318800 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:27.318753    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz" podUID="5c887260-5348-43db-a90c-98c687c08f8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jun 15 06:53:37.323204 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:37.323179    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"
Jun 15 06:53:44.399165 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:44.399127    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"]
Jun 15 06:53:44.399603 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:44.399371    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz" podUID="5c887260-5348-43db-a90c-98c687c08f8c" containerName="kserve-container" containerID="cri-o://684e2c12f6dfe151adf685fcee78b6a07696b036caa96ec25630bff133b72978" gracePeriod=30
Jun 15 06:53:44.399603 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:44.399431    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz" podUID="5c887260-5348-43db-a90c-98c687c08f8c" containerName="kube-rbac-proxy" containerID="cri-o://c1fcb3681313b7051ebf4d0bc346e7420e5ba8a5c058512fdab30cdacae353a2" gracePeriod=30
Jun 15 06:53:44.512405 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:44.512348    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz" podUID="5c887260-5348-43db-a90c-98c687c08f8c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.60:8643/healthz\": dial tcp 10.134.0.60:8643: connect: connection refused"
Jun 15 06:53:44.913565 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:44.913531    2572 generic.go:356] "Generic (PLEG): container finished" podID="5c887260-5348-43db-a90c-98c687c08f8c" containerID="c1fcb3681313b7051ebf4d0bc346e7420e5ba8a5c058512fdab30cdacae353a2" exitCode=2
Jun 15 06:53:44.913721 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:44.913569    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz" event={"ID":"5c887260-5348-43db-a90c-98c687c08f8c","Type":"ContainerDied","Data":"c1fcb3681313b7051ebf4d0bc346e7420e5ba8a5c058512fdab30cdacae353a2"}
Jun 15 06:53:45.558450 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:45.558407    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw"]
Jun 15 06:53:45.558971 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:45.558951    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8620513-f166-45c8-aefe-0aa14fe5d0b6" containerName="storage-initializer"
Jun 15 06:53:45.559050 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:45.558975    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8620513-f166-45c8-aefe-0aa14fe5d0b6" containerName="storage-initializer"
Jun 15 06:53:45.559050 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:45.558989    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8620513-f166-45c8-aefe-0aa14fe5d0b6" containerName="kserve-container"
Jun 15 06:53:45.559050 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:45.558998    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8620513-f166-45c8-aefe-0aa14fe5d0b6" containerName="kserve-container"
Jun 15 06:53:45.559050 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:45.559030    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8620513-f166-45c8-aefe-0aa14fe5d0b6" containerName="kube-rbac-proxy"
Jun 15 06:53:45.559050 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:45.559039    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8620513-f166-45c8-aefe-0aa14fe5d0b6" containerName="kube-rbac-proxy"
Jun 15 06:53:45.559307 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:45.559122    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8620513-f166-45c8-aefe-0aa14fe5d0b6" containerName="kserve-container"
Jun 15 06:53:45.559307 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:45.559137    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8620513-f166-45c8-aefe-0aa14fe5d0b6" containerName="kube-rbac-proxy"
Jun 15 06:53:45.562374 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:45.562337    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw"
Jun 15 06:53:45.564604 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:45.564584    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-predictor-serving-cert\""
Jun 15 06:53:45.564718 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:45.564639    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-fail-kube-rbac-proxy-sar-config\""
Jun 15 06:53:45.572022 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:45.571994    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw"]
Jun 15 06:53:45.720663 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:45.720627    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0706b4d1-7be2-410d-bd3c-c579fa60ba05-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw\" (UID: \"0706b4d1-7be2-410d-bd3c-c579fa60ba05\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw"
Jun 15 06:53:45.720839 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:45.720675    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0706b4d1-7be2-410d-bd3c-c579fa60ba05-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw\" (UID: \"0706b4d1-7be2-410d-bd3c-c579fa60ba05\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw"
Jun 15 06:53:45.720839 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:45.720778    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0706b4d1-7be2-410d-bd3c-c579fa60ba05-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw\" (UID: \"0706b4d1-7be2-410d-bd3c-c579fa60ba05\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw"
Jun 15 06:53:45.720839 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:45.720818    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f5bz\" (UniqueName: \"kubernetes.io/projected/0706b4d1-7be2-410d-bd3c-c579fa60ba05-kube-api-access-6f5bz\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw\" (UID: \"0706b4d1-7be2-410d-bd3c-c579fa60ba05\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw"
Jun 15 06:53:45.822349 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:45.822253    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0706b4d1-7be2-410d-bd3c-c579fa60ba05-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw\" (UID: \"0706b4d1-7be2-410d-bd3c-c579fa60ba05\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw"
Jun 15 06:53:45.822349 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:45.822301    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0706b4d1-7be2-410d-bd3c-c579fa60ba05-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw\" (UID: \"0706b4d1-7be2-410d-bd3c-c579fa60ba05\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw"
Jun 15 06:53:45.822620 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:45.822353    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0706b4d1-7be2-410d-bd3c-c579fa60ba05-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw\" (UID: \"0706b4d1-7be2-410d-bd3c-c579fa60ba05\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw"
Jun 15 06:53:45.822620 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:45.822415    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f5bz\" (UniqueName: \"kubernetes.io/projected/0706b4d1-7be2-410d-bd3c-c579fa60ba05-kube-api-access-6f5bz\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw\" (UID: \"0706b4d1-7be2-410d-bd3c-c579fa60ba05\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw"
Jun 15 06:53:45.822835 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:45.822803    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0706b4d1-7be2-410d-bd3c-c579fa60ba05-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw\" (UID: \"0706b4d1-7be2-410d-bd3c-c579fa60ba05\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw"
Jun 15 06:53:45.823009 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:45.822992    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0706b4d1-7be2-410d-bd3c-c579fa60ba05-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw\" (UID: \"0706b4d1-7be2-410d-bd3c-c579fa60ba05\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw"
Jun 15 06:53:45.824901 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:45.824879    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0706b4d1-7be2-410d-bd3c-c579fa60ba05-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw\" (UID: \"0706b4d1-7be2-410d-bd3c-c579fa60ba05\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw"
Jun 15 06:53:45.832191 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:45.832164    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f5bz\" (UniqueName: \"kubernetes.io/projected/0706b4d1-7be2-410d-bd3c-c579fa60ba05-kube-api-access-6f5bz\") pod \"isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw\" (UID: \"0706b4d1-7be2-410d-bd3c-c579fa60ba05\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw"
Jun 15 06:53:45.874063 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:45.874023    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw"
Jun 15 06:53:46.011035 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:46.011010    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw"]
Jun 15 06:53:46.013051 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:53:46.013023    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0706b4d1_7be2_410d_bd3c_c579fa60ba05.slice/crio-a2fe5fb462622d8f9dadf8736d734ec5b49dd06aac59e80dbb9f56d7a8410925 WatchSource:0}: Error finding container a2fe5fb462622d8f9dadf8736d734ec5b49dd06aac59e80dbb9f56d7a8410925: Status 404 returned error can't find the container with id a2fe5fb462622d8f9dadf8736d734ec5b49dd06aac59e80dbb9f56d7a8410925
Jun 15 06:53:46.924704 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:46.924669    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw" event={"ID":"0706b4d1-7be2-410d-bd3c-c579fa60ba05","Type":"ContainerStarted","Data":"3a0017f129da416915df40a18bc65398f94aeba83cf694713a36d5f2ed1e71a0"}
Jun 15 06:53:46.925154 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:46.924736    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw" event={"ID":"0706b4d1-7be2-410d-bd3c-c579fa60ba05","Type":"ContainerStarted","Data":"a2fe5fb462622d8f9dadf8736d734ec5b49dd06aac59e80dbb9f56d7a8410925"}
Jun 15 06:53:47.319060 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:47.318954    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz" podUID="5c887260-5348-43db-a90c-98c687c08f8c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.60:8080: connect: connection refused"
Jun 15 06:53:49.144639 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.144616    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"
Jun 15 06:53:49.149029 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.149011    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c887260-5348-43db-a90c-98c687c08f8c-cabundle-cert\") pod \"5c887260-5348-43db-a90c-98c687c08f8c\" (UID: \"5c887260-5348-43db-a90c-98c687c08f8c\") "
Jun 15 06:53:49.149158 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.149062    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c887260-5348-43db-a90c-98c687c08f8c-kserve-provision-location\") pod \"5c887260-5348-43db-a90c-98c687c08f8c\" (UID: \"5c887260-5348-43db-a90c-98c687c08f8c\") "
Jun 15 06:53:49.149158 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.149098    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c887260-5348-43db-a90c-98c687c08f8c-proxy-tls\") pod \"5c887260-5348-43db-a90c-98c687c08f8c\" (UID: \"5c887260-5348-43db-a90c-98c687c08f8c\") "
Jun 15 06:53:49.149158 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.149136    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c887260-5348-43db-a90c-98c687c08f8c-kube-rbac-proxy-sar-config\") pod \"5c887260-5348-43db-a90c-98c687c08f8c\" (UID: \"5c887260-5348-43db-a90c-98c687c08f8c\") "
Jun 15 06:53:49.149328 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.149162    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7gzw\" (UniqueName: \"kubernetes.io/projected/5c887260-5348-43db-a90c-98c687c08f8c-kube-api-access-t7gzw\") pod \"5c887260-5348-43db-a90c-98c687c08f8c\" (UID: \"5c887260-5348-43db-a90c-98c687c08f8c\") "
Jun 15 06:53:49.149450 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.149428    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c887260-5348-43db-a90c-98c687c08f8c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5c887260-5348-43db-a90c-98c687c08f8c" (UID: "5c887260-5348-43db-a90c-98c687c08f8c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:53:49.149450 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.149436    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c887260-5348-43db-a90c-98c687c08f8c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5c887260-5348-43db-a90c-98c687c08f8c" (UID: "5c887260-5348-43db-a90c-98c687c08f8c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:53:49.149568 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.149485    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c887260-5348-43db-a90c-98c687c08f8c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5c887260-5348-43db-a90c-98c687c08f8c" (UID: "5c887260-5348-43db-a90c-98c687c08f8c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:53:49.149568 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.149526    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5c887260-5348-43db-a90c-98c687c08f8c-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:53:49.149568 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.149540    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5c887260-5348-43db-a90c-98c687c08f8c-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:53:49.151191 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.151173    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c887260-5348-43db-a90c-98c687c08f8c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5c887260-5348-43db-a90c-98c687c08f8c" (UID: "5c887260-5348-43db-a90c-98c687c08f8c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:53:49.151269 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.151202    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c887260-5348-43db-a90c-98c687c08f8c-kube-api-access-t7gzw" (OuterVolumeSpecName: "kube-api-access-t7gzw") pod "5c887260-5348-43db-a90c-98c687c08f8c" (UID: "5c887260-5348-43db-a90c-98c687c08f8c"). InnerVolumeSpecName "kube-api-access-t7gzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:53:49.250658 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.250566    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5c887260-5348-43db-a90c-98c687c08f8c-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:53:49.250658 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.250598    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5c887260-5348-43db-a90c-98c687c08f8c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:53:49.250658 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.250611    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7gzw\" (UniqueName: \"kubernetes.io/projected/5c887260-5348-43db-a90c-98c687c08f8c-kube-api-access-t7gzw\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:53:49.940446 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.940410    2572 generic.go:356] "Generic (PLEG): container finished" podID="5c887260-5348-43db-a90c-98c687c08f8c" containerID="684e2c12f6dfe151adf685fcee78b6a07696b036caa96ec25630bff133b72978" exitCode=0
Jun 15 06:53:49.940624 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.940467    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz" event={"ID":"5c887260-5348-43db-a90c-98c687c08f8c","Type":"ContainerDied","Data":"684e2c12f6dfe151adf685fcee78b6a07696b036caa96ec25630bff133b72978"}
Jun 15 06:53:49.940624 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.940479    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"
Jun 15 06:53:49.940624 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.940516    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz" event={"ID":"5c887260-5348-43db-a90c-98c687c08f8c","Type":"ContainerDied","Data":"60f656c0f33343c7ded1e73e47a2700955bfa10523d6e5d495b4a8f2f0007df7"}
Jun 15 06:53:49.940624 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.940542    2572 scope.go:117] "RemoveContainer" containerID="c1fcb3681313b7051ebf4d0bc346e7420e5ba8a5c058512fdab30cdacae353a2"
Jun 15 06:53:49.949107 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.949087    2572 scope.go:117] "RemoveContainer" containerID="684e2c12f6dfe151adf685fcee78b6a07696b036caa96ec25630bff133b72978"
Jun 15 06:53:49.956388 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.956372    2572 scope.go:117] "RemoveContainer" containerID="8f1697aaed5f37122ac5f02987e3262c8fc6efdb2271030d33c782a5e735e560"
Jun 15 06:53:49.962601 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.962579    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"]
Jun 15 06:53:49.965456 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.965433    2572 scope.go:117] "RemoveContainer" containerID="c1fcb3681313b7051ebf4d0bc346e7420e5ba8a5c058512fdab30cdacae353a2"
Jun 15 06:53:49.965911 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:53:49.965878    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1fcb3681313b7051ebf4d0bc346e7420e5ba8a5c058512fdab30cdacae353a2\": container with ID starting with c1fcb3681313b7051ebf4d0bc346e7420e5ba8a5c058512fdab30cdacae353a2 not found: ID does not exist" containerID="c1fcb3681313b7051ebf4d0bc346e7420e5ba8a5c058512fdab30cdacae353a2"
Jun 15 06:53:49.966005 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.965917    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1fcb3681313b7051ebf4d0bc346e7420e5ba8a5c058512fdab30cdacae353a2"} err="failed to get container status \"c1fcb3681313b7051ebf4d0bc346e7420e5ba8a5c058512fdab30cdacae353a2\": rpc error: code = NotFound desc = could not find container \"c1fcb3681313b7051ebf4d0bc346e7420e5ba8a5c058512fdab30cdacae353a2\": container with ID starting with c1fcb3681313b7051ebf4d0bc346e7420e5ba8a5c058512fdab30cdacae353a2 not found: ID does not exist"
Jun 15 06:53:49.966005 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.965947    2572 scope.go:117] "RemoveContainer" containerID="684e2c12f6dfe151adf685fcee78b6a07696b036caa96ec25630bff133b72978"
Jun 15 06:53:49.966285 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:53:49.966259    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"684e2c12f6dfe151adf685fcee78b6a07696b036caa96ec25630bff133b72978\": container with ID starting with 684e2c12f6dfe151adf685fcee78b6a07696b036caa96ec25630bff133b72978 not found: ID does not exist" containerID="684e2c12f6dfe151adf685fcee78b6a07696b036caa96ec25630bff133b72978"
Jun 15 06:53:49.966410 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.966291    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"684e2c12f6dfe151adf685fcee78b6a07696b036caa96ec25630bff133b72978"} err="failed to get container status \"684e2c12f6dfe151adf685fcee78b6a07696b036caa96ec25630bff133b72978\": rpc error: code = NotFound desc = could not find container \"684e2c12f6dfe151adf685fcee78b6a07696b036caa96ec25630bff133b72978\": container with ID starting with 684e2c12f6dfe151adf685fcee78b6a07696b036caa96ec25630bff133b72978 not found: ID does not exist"
Jun 15 06:53:49.966410 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.966307    2572 scope.go:117] "RemoveContainer" containerID="8f1697aaed5f37122ac5f02987e3262c8fc6efdb2271030d33c782a5e735e560"
Jun 15 06:53:49.966641 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:53:49.966623    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f1697aaed5f37122ac5f02987e3262c8fc6efdb2271030d33c782a5e735e560\": container with ID starting with 8f1697aaed5f37122ac5f02987e3262c8fc6efdb2271030d33c782a5e735e560 not found: ID does not exist" containerID="8f1697aaed5f37122ac5f02987e3262c8fc6efdb2271030d33c782a5e735e560"
Jun 15 06:53:49.966695 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.966662    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f1697aaed5f37122ac5f02987e3262c8fc6efdb2271030d33c782a5e735e560"} err="failed to get container status \"8f1697aaed5f37122ac5f02987e3262c8fc6efdb2271030d33c782a5e735e560\": rpc error: code = NotFound desc = could not find container \"8f1697aaed5f37122ac5f02987e3262c8fc6efdb2271030d33c782a5e735e560\": container with ID starting with 8f1697aaed5f37122ac5f02987e3262c8fc6efdb2271030d33c782a5e735e560 not found: ID does not exist"
Jun 15 06:53:49.967324 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:49.967307    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-676dfc7d88-p9hzz"]
Jun 15 06:53:51.324139 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:51.324058    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c887260-5348-43db-a90c-98c687c08f8c" path="/var/lib/kubelet/pods/5c887260-5348-43db-a90c-98c687c08f8c/volumes"
Jun 15 06:53:51.951465 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:51.951439    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw_0706b4d1-7be2-410d-bd3c-c579fa60ba05/storage-initializer/0.log"
Jun 15 06:53:51.951645 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:51.951489    2572 generic.go:356] "Generic (PLEG): container finished" podID="0706b4d1-7be2-410d-bd3c-c579fa60ba05" containerID="3a0017f129da416915df40a18bc65398f94aeba83cf694713a36d5f2ed1e71a0" exitCode=1
Jun 15 06:53:51.951645 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:51.951516    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw" event={"ID":"0706b4d1-7be2-410d-bd3c-c579fa60ba05","Type":"ContainerDied","Data":"3a0017f129da416915df40a18bc65398f94aeba83cf694713a36d5f2ed1e71a0"}
Jun 15 06:53:52.958188 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:52.958158    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw_0706b4d1-7be2-410d-bd3c-c579fa60ba05/storage-initializer/0.log"
Jun 15 06:53:52.958583 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:52.958229    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw" event={"ID":"0706b4d1-7be2-410d-bd3c-c579fa60ba05","Type":"ContainerStarted","Data":"3da71fe734064042882f25bbbca265791a8367476891493216b04c0a16578f54"}
Jun 15 06:53:55.552102 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:55.552070    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw"]
Jun 15 06:53:55.552482 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:55.552283    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw" podUID="0706b4d1-7be2-410d-bd3c-c579fa60ba05" containerName="storage-initializer" containerID="cri-o://3da71fe734064042882f25bbbca265791a8367476891493216b04c0a16578f54" gracePeriod=30
Jun 15 06:53:56.192725 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.192700    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw_0706b4d1-7be2-410d-bd3c-c579fa60ba05/storage-initializer/1.log"
Jun 15 06:53:56.193076 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.193063    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw_0706b4d1-7be2-410d-bd3c-c579fa60ba05/storage-initializer/0.log"
Jun 15 06:53:56.195273 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.195258    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw"
Jun 15 06:53:56.210891 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.210867    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0706b4d1-7be2-410d-bd3c-c579fa60ba05-kserve-provision-location\") pod \"0706b4d1-7be2-410d-bd3c-c579fa60ba05\" (UID: \"0706b4d1-7be2-410d-bd3c-c579fa60ba05\") "
Jun 15 06:53:56.210971 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.210903    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0706b4d1-7be2-410d-bd3c-c579fa60ba05-kube-rbac-proxy-sar-config\") pod \"0706b4d1-7be2-410d-bd3c-c579fa60ba05\" (UID: \"0706b4d1-7be2-410d-bd3c-c579fa60ba05\") "
Jun 15 06:53:56.210971 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.210969    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6f5bz\" (UniqueName: \"kubernetes.io/projected/0706b4d1-7be2-410d-bd3c-c579fa60ba05-kube-api-access-6f5bz\") pod \"0706b4d1-7be2-410d-bd3c-c579fa60ba05\" (UID: \"0706b4d1-7be2-410d-bd3c-c579fa60ba05\") "
Jun 15 06:53:56.211072 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.211050    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0706b4d1-7be2-410d-bd3c-c579fa60ba05-proxy-tls\") pod \"0706b4d1-7be2-410d-bd3c-c579fa60ba05\" (UID: \"0706b4d1-7be2-410d-bd3c-c579fa60ba05\") "
Jun 15 06:53:56.211184 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.211161    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0706b4d1-7be2-410d-bd3c-c579fa60ba05-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0706b4d1-7be2-410d-bd3c-c579fa60ba05" (UID: "0706b4d1-7be2-410d-bd3c-c579fa60ba05"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:53:56.211281 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.211257    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0706b4d1-7be2-410d-bd3c-c579fa60ba05-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0706b4d1-7be2-410d-bd3c-c579fa60ba05" (UID: "0706b4d1-7be2-410d-bd3c-c579fa60ba05"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:53:56.211372 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.211342    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0706b4d1-7be2-410d-bd3c-c579fa60ba05-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:53:56.211434 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.211380    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0706b4d1-7be2-410d-bd3c-c579fa60ba05-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:53:56.213046 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.213024    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0706b4d1-7be2-410d-bd3c-c579fa60ba05-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0706b4d1-7be2-410d-bd3c-c579fa60ba05" (UID: "0706b4d1-7be2-410d-bd3c-c579fa60ba05"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:53:56.213132 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.213071    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0706b4d1-7be2-410d-bd3c-c579fa60ba05-kube-api-access-6f5bz" (OuterVolumeSpecName: "kube-api-access-6f5bz") pod "0706b4d1-7be2-410d-bd3c-c579fa60ba05" (UID: "0706b4d1-7be2-410d-bd3c-c579fa60ba05"). InnerVolumeSpecName "kube-api-access-6f5bz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:53:56.312044 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.311951    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0706b4d1-7be2-410d-bd3c-c579fa60ba05-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:53:56.312044 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.311988    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6f5bz\" (UniqueName: \"kubernetes.io/projected/0706b4d1-7be2-410d-bd3c-c579fa60ba05-kube-api-access-6f5bz\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:53:56.588847 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.588772    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"]
Jun 15 06:53:56.589209 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.589146    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c887260-5348-43db-a90c-98c687c08f8c" containerName="storage-initializer"
Jun 15 06:53:56.589209 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.589160    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c887260-5348-43db-a90c-98c687c08f8c" containerName="storage-initializer"
Jun 15 06:53:56.589209 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.589173    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c887260-5348-43db-a90c-98c687c08f8c" containerName="kserve-container"
Jun 15 06:53:56.589209 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.589178    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c887260-5348-43db-a90c-98c687c08f8c" containerName="kserve-container"
Jun 15 06:53:56.589209 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.589194    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0706b4d1-7be2-410d-bd3c-c579fa60ba05" containerName="storage-initializer"
Jun 15 06:53:56.589209 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.589201    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0706b4d1-7be2-410d-bd3c-c579fa60ba05" containerName="storage-initializer"
Jun 15 06:53:56.589209 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.589208    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c887260-5348-43db-a90c-98c687c08f8c" containerName="kube-rbac-proxy"
Jun 15 06:53:56.589209 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.589213    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c887260-5348-43db-a90c-98c687c08f8c" containerName="kube-rbac-proxy"
Jun 15 06:53:56.589495 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.589219    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0706b4d1-7be2-410d-bd3c-c579fa60ba05" containerName="storage-initializer"
Jun 15 06:53:56.589495 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.589224    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="0706b4d1-7be2-410d-bd3c-c579fa60ba05" containerName="storage-initializer"
Jun 15 06:53:56.589495 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.589298    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0706b4d1-7be2-410d-bd3c-c579fa60ba05" containerName="storage-initializer"
Jun 15 06:53:56.589495 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.589306    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c887260-5348-43db-a90c-98c687c08f8c" containerName="kube-rbac-proxy"
Jun 15 06:53:56.589495 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.589315    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c887260-5348-43db-a90c-98c687c08f8c" containerName="kserve-container"
Jun 15 06:53:56.589495 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.589431    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="0706b4d1-7be2-410d-bd3c-c579fa60ba05" containerName="storage-initializer"
Jun 15 06:53:56.592691 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.592671    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"
Jun 15 06:53:56.594718 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.594695    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Jun 15 06:53:56.594804 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.594695    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jun 15 06:53:56.594804 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.594759    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Jun 15 06:53:56.601900 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.601877    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"]
Jun 15 06:53:56.614943 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.614919    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4z42\" (UniqueName: \"kubernetes.io/projected/97ee3963-34fe-4f7d-8567-4d4a4f63a538-kube-api-access-l4z42\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v\" (UID: \"97ee3963-34fe-4f7d-8567-4d4a4f63a538\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"
Jun 15 06:53:56.615047 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.614953    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97ee3963-34fe-4f7d-8567-4d4a4f63a538-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v\" (UID: \"97ee3963-34fe-4f7d-8567-4d4a4f63a538\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"
Jun 15 06:53:56.615047 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.615011    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97ee3963-34fe-4f7d-8567-4d4a4f63a538-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v\" (UID: \"97ee3963-34fe-4f7d-8567-4d4a4f63a538\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"
Jun 15 06:53:56.615214 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.615081    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97ee3963-34fe-4f7d-8567-4d4a4f63a538-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v\" (UID: \"97ee3963-34fe-4f7d-8567-4d4a4f63a538\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"
Jun 15 06:53:56.615214 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.615102    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97ee3963-34fe-4f7d-8567-4d4a4f63a538-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v\" (UID: \"97ee3963-34fe-4f7d-8567-4d4a4f63a538\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"
Jun 15 06:53:56.715791 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.715752    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4z42\" (UniqueName: \"kubernetes.io/projected/97ee3963-34fe-4f7d-8567-4d4a4f63a538-kube-api-access-l4z42\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v\" (UID: \"97ee3963-34fe-4f7d-8567-4d4a4f63a538\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"
Jun 15 06:53:56.715791 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.715790    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97ee3963-34fe-4f7d-8567-4d4a4f63a538-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v\" (UID: \"97ee3963-34fe-4f7d-8567-4d4a4f63a538\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"
Jun 15 06:53:56.716038 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.715853    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97ee3963-34fe-4f7d-8567-4d4a4f63a538-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v\" (UID: \"97ee3963-34fe-4f7d-8567-4d4a4f63a538\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"
Jun 15 06:53:56.716038 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.715900    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97ee3963-34fe-4f7d-8567-4d4a4f63a538-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v\" (UID: \"97ee3963-34fe-4f7d-8567-4d4a4f63a538\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"
Jun 15 06:53:56.716038 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.715921    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97ee3963-34fe-4f7d-8567-4d4a4f63a538-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v\" (UID: \"97ee3963-34fe-4f7d-8567-4d4a4f63a538\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"
Jun 15 06:53:56.716038 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:53:56.716025    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Jun 15 06:53:56.716251 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:53:56.716088    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97ee3963-34fe-4f7d-8567-4d4a4f63a538-proxy-tls podName:97ee3963-34fe-4f7d-8567-4d4a4f63a538 nodeName:}" failed. No retries permitted until 2026-06-15 06:53:57.216067269 +0000 UTC m=+3952.364247250 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/97ee3963-34fe-4f7d-8567-4d4a4f63a538-proxy-tls") pod "isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v" (UID: "97ee3963-34fe-4f7d-8567-4d4a4f63a538") : secret "isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert" not found
Jun 15 06:53:56.716322 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.716295    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97ee3963-34fe-4f7d-8567-4d4a4f63a538-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v\" (UID: \"97ee3963-34fe-4f7d-8567-4d4a4f63a538\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"
Jun 15 06:53:56.716521 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.716502    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97ee3963-34fe-4f7d-8567-4d4a4f63a538-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v\" (UID: \"97ee3963-34fe-4f7d-8567-4d4a4f63a538\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"
Jun 15 06:53:56.716601 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.716585    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97ee3963-34fe-4f7d-8567-4d4a4f63a538-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v\" (UID: \"97ee3963-34fe-4f7d-8567-4d4a4f63a538\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"
Jun 15 06:53:56.725609 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.725590    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4z42\" (UniqueName: \"kubernetes.io/projected/97ee3963-34fe-4f7d-8567-4d4a4f63a538-kube-api-access-l4z42\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v\" (UID: \"97ee3963-34fe-4f7d-8567-4d4a4f63a538\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"
Jun 15 06:53:56.976010 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.975981    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw_0706b4d1-7be2-410d-bd3c-c579fa60ba05/storage-initializer/1.log"
Jun 15 06:53:56.976425 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.976409    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw_0706b4d1-7be2-410d-bd3c-c579fa60ba05/storage-initializer/0.log"
Jun 15 06:53:56.976522 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.976443    2572 generic.go:356] "Generic (PLEG): container finished" podID="0706b4d1-7be2-410d-bd3c-c579fa60ba05" containerID="3da71fe734064042882f25bbbca265791a8367476891493216b04c0a16578f54" exitCode=1
Jun 15 06:53:56.976522 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.976501    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw"
Jun 15 06:53:56.976601 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.976513    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw" event={"ID":"0706b4d1-7be2-410d-bd3c-c579fa60ba05","Type":"ContainerDied","Data":"3da71fe734064042882f25bbbca265791a8367476891493216b04c0a16578f54"}
Jun 15 06:53:56.976601 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.976574    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw" event={"ID":"0706b4d1-7be2-410d-bd3c-c579fa60ba05","Type":"ContainerDied","Data":"a2fe5fb462622d8f9dadf8736d734ec5b49dd06aac59e80dbb9f56d7a8410925"}
Jun 15 06:53:56.976667 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.976603    2572 scope.go:117] "RemoveContainer" containerID="3da71fe734064042882f25bbbca265791a8367476891493216b04c0a16578f54"
Jun 15 06:53:56.985574 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.985557    2572 scope.go:117] "RemoveContainer" containerID="3a0017f129da416915df40a18bc65398f94aeba83cf694713a36d5f2ed1e71a0"
Jun 15 06:53:56.993119 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.993103    2572 scope.go:117] "RemoveContainer" containerID="3da71fe734064042882f25bbbca265791a8367476891493216b04c0a16578f54"
Jun 15 06:53:56.993379 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:53:56.993341    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3da71fe734064042882f25bbbca265791a8367476891493216b04c0a16578f54\": container with ID starting with 3da71fe734064042882f25bbbca265791a8367476891493216b04c0a16578f54 not found: ID does not exist" containerID="3da71fe734064042882f25bbbca265791a8367476891493216b04c0a16578f54"
Jun 15 06:53:56.993442 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.993380    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3da71fe734064042882f25bbbca265791a8367476891493216b04c0a16578f54"} err="failed to get container status \"3da71fe734064042882f25bbbca265791a8367476891493216b04c0a16578f54\": rpc error: code = NotFound desc = could not find container \"3da71fe734064042882f25bbbca265791a8367476891493216b04c0a16578f54\": container with ID starting with 3da71fe734064042882f25bbbca265791a8367476891493216b04c0a16578f54 not found: ID does not exist"
Jun 15 06:53:56.993442 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.993398    2572 scope.go:117] "RemoveContainer" containerID="3a0017f129da416915df40a18bc65398f94aeba83cf694713a36d5f2ed1e71a0"
Jun 15 06:53:56.993637 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:53:56.993619    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a0017f129da416915df40a18bc65398f94aeba83cf694713a36d5f2ed1e71a0\": container with ID starting with 3a0017f129da416915df40a18bc65398f94aeba83cf694713a36d5f2ed1e71a0 not found: ID does not exist" containerID="3a0017f129da416915df40a18bc65398f94aeba83cf694713a36d5f2ed1e71a0"
Jun 15 06:53:56.993691 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:56.993645    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a0017f129da416915df40a18bc65398f94aeba83cf694713a36d5f2ed1e71a0"} err="failed to get container status \"3a0017f129da416915df40a18bc65398f94aeba83cf694713a36d5f2ed1e71a0\": rpc error: code = NotFound desc = could not find container \"3a0017f129da416915df40a18bc65398f94aeba83cf694713a36d5f2ed1e71a0\": container with ID starting with 3a0017f129da416915df40a18bc65398f94aeba83cf694713a36d5f2ed1e71a0 not found: ID does not exist"
Jun 15 06:53:57.015013 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:57.014983    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw"]
Jun 15 06:53:57.020000 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:57.019971    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-fail-predictor-5544cfc5b-4dkxw"]
Jun 15 06:53:57.220257 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:57.220217    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97ee3963-34fe-4f7d-8567-4d4a4f63a538-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v\" (UID: \"97ee3963-34fe-4f7d-8567-4d4a4f63a538\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"
Jun 15 06:53:57.222705 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:57.222684    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97ee3963-34fe-4f7d-8567-4d4a4f63a538-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v\" (UID: \"97ee3963-34fe-4f7d-8567-4d4a4f63a538\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"
Jun 15 06:53:57.323599 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:57.323525    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0706b4d1-7be2-410d-bd3c-c579fa60ba05" path="/var/lib/kubelet/pods/0706b4d1-7be2-410d-bd3c-c579fa60ba05/volumes"
Jun 15 06:53:57.504477 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:57.504452    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"
Jun 15 06:53:57.638040 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:57.637993    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"]
Jun 15 06:53:57.640237 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:53:57.640211    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97ee3963_34fe_4f7d_8567_4d4a4f63a538.slice/crio-f4aceaf607b7f35b81bacddfb30561b34df06697d1944bb9502cdf101af39097 WatchSource:0}: Error finding container f4aceaf607b7f35b81bacddfb30561b34df06697d1944bb9502cdf101af39097: Status 404 returned error can't find the container with id f4aceaf607b7f35b81bacddfb30561b34df06697d1944bb9502cdf101af39097
Jun 15 06:53:57.983981 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:57.983947    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v" event={"ID":"97ee3963-34fe-4f7d-8567-4d4a4f63a538","Type":"ContainerStarted","Data":"f5996bf2624966bea89558c989e0ed795615d7f3f55d022b7d8e6b3e780424a8"}
Jun 15 06:53:57.984138 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:57.984000    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v" event={"ID":"97ee3963-34fe-4f7d-8567-4d4a4f63a538","Type":"ContainerStarted","Data":"f4aceaf607b7f35b81bacddfb30561b34df06697d1944bb9502cdf101af39097"}
Jun 15 06:53:58.990341 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:58.990256    2572 generic.go:356] "Generic (PLEG): container finished" podID="97ee3963-34fe-4f7d-8567-4d4a4f63a538" containerID="f5996bf2624966bea89558c989e0ed795615d7f3f55d022b7d8e6b3e780424a8" exitCode=0
Jun 15 06:53:58.990770 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:58.990348    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v" event={"ID":"97ee3963-34fe-4f7d-8567-4d4a4f63a538","Type":"ContainerDied","Data":"f5996bf2624966bea89558c989e0ed795615d7f3f55d022b7d8e6b3e780424a8"}
Jun 15 06:53:59.997490 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:59.997456    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v" event={"ID":"97ee3963-34fe-4f7d-8567-4d4a4f63a538","Type":"ContainerStarted","Data":"13f1a3f378f010be54c3b9ac66572089f7d494340e64cadefc54d90961bbddef"}
Jun 15 06:53:59.997861 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:59.997520    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v" event={"ID":"97ee3963-34fe-4f7d-8567-4d4a4f63a538","Type":"ContainerStarted","Data":"b95bbe7efc2b80f4dfa186e54575854600ff9489215fe3cc0c971d47c95c56f7"}
Jun 15 06:53:59.997861 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:53:59.997606    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"
Jun 15 06:54:00.018920 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:54:00.018869    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v" podStartSLOduration=3.008813842 podStartE2EDuration="4.018855887s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.010042045s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:54:00.016797842 +0000 UTC m=+3955.164977839" watchObservedRunningTime="2026-06-15 06:54:00.018855887 +0000 UTC m=+3955.167035883"
Jun 15 06:54:01.002349 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:54:01.002319    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"
Jun 15 06:54:01.003759 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:54:01.003732    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v" podUID="97ee3963-34fe-4f7d-8567-4d4a4f63a538" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Jun 15 06:54:02.007820 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:54:02.007780    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v" podUID="97ee3963-34fe-4f7d-8567-4d4a4f63a538" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Jun 15 06:54:07.011811 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:54:07.011783    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"
Jun 15 06:54:07.012313 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:54:07.012285    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v" podUID="97ee3963-34fe-4f7d-8567-4d4a4f63a538" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Jun 15 06:54:17.012453 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:54:17.012409    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v" podUID="97ee3963-34fe-4f7d-8567-4d4a4f63a538" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Jun 15 06:54:27.012610 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:54:27.012573    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v" podUID="97ee3963-34fe-4f7d-8567-4d4a4f63a538" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Jun 15 06:54:37.013204 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:54:37.013164    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v" podUID="97ee3963-34fe-4f7d-8567-4d4a4f63a538" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Jun 15 06:54:47.013002 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:54:47.012960    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v" podUID="97ee3963-34fe-4f7d-8567-4d4a4f63a538" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Jun 15 06:54:57.012537 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:54:57.012497    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v" podUID="97ee3963-34fe-4f7d-8567-4d4a4f63a538" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Jun 15 06:55:07.012864 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:07.012826    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v" podUID="97ee3963-34fe-4f7d-8567-4d4a4f63a538" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Jun 15 06:55:17.013028 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:17.012943    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"
Jun 15 06:55:26.665114 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:26.665073    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"]
Jun 15 06:55:26.665541 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:26.665378    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v" podUID="97ee3963-34fe-4f7d-8567-4d4a4f63a538" containerName="kserve-container" containerID="cri-o://b95bbe7efc2b80f4dfa186e54575854600ff9489215fe3cc0c971d47c95c56f7" gracePeriod=30
Jun 15 06:55:26.665541 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:26.665414    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v" podUID="97ee3963-34fe-4f7d-8567-4d4a4f63a538" containerName="kube-rbac-proxy" containerID="cri-o://13f1a3f378f010be54c3b9ac66572089f7d494340e64cadefc54d90961bbddef" gracePeriod=30
Jun 15 06:55:27.008581 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:27.008493    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v" podUID="97ee3963-34fe-4f7d-8567-4d4a4f63a538" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.62:8643/healthz\": dial tcp 10.134.0.62:8643: connect: connection refused"
Jun 15 06:55:27.012866 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:27.012839    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v" podUID="97ee3963-34fe-4f7d-8567-4d4a4f63a538" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.62:8080: connect: connection refused"
Jun 15 06:55:27.399336 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:27.399302    2572 generic.go:356] "Generic (PLEG): container finished" podID="97ee3963-34fe-4f7d-8567-4d4a4f63a538" containerID="13f1a3f378f010be54c3b9ac66572089f7d494340e64cadefc54d90961bbddef" exitCode=2
Jun 15 06:55:27.399522 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:27.399393    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v" event={"ID":"97ee3963-34fe-4f7d-8567-4d4a4f63a538","Type":"ContainerDied","Data":"13f1a3f378f010be54c3b9ac66572089f7d494340e64cadefc54d90961bbddef"}
Jun 15 06:55:27.775242 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:27.775163    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68"]
Jun 15 06:55:27.778805 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:27.778784    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68"
Jun 15 06:55:27.780661 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:27.780642    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Jun 15 06:55:27.780895 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:27.780882    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Jun 15 06:55:27.789819 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:27.789799    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68"]
Jun 15 06:55:27.821501 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:27.821470    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qqmq\" (UniqueName: \"kubernetes.io/projected/ec8810d0-da8c-4f16-a509-a50fe83ae4b2-kube-api-access-6qqmq\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68\" (UID: \"ec8810d0-da8c-4f16-a509-a50fe83ae4b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68"
Jun 15 06:55:27.821643 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:27.821520    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec8810d0-da8c-4f16-a509-a50fe83ae4b2-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68\" (UID: \"ec8810d0-da8c-4f16-a509-a50fe83ae4b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68"
Jun 15 06:55:27.821643 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:27.821587    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec8810d0-da8c-4f16-a509-a50fe83ae4b2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68\" (UID: \"ec8810d0-da8c-4f16-a509-a50fe83ae4b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68"
Jun 15 06:55:27.821643 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:27.821619    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec8810d0-da8c-4f16-a509-a50fe83ae4b2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68\" (UID: \"ec8810d0-da8c-4f16-a509-a50fe83ae4b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68"
Jun 15 06:55:27.922042 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:27.922003    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec8810d0-da8c-4f16-a509-a50fe83ae4b2-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68\" (UID: \"ec8810d0-da8c-4f16-a509-a50fe83ae4b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68"
Jun 15 06:55:27.922237 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:27.922083    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec8810d0-da8c-4f16-a509-a50fe83ae4b2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68\" (UID: \"ec8810d0-da8c-4f16-a509-a50fe83ae4b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68"
Jun 15 06:55:27.922237 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:27.922132    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec8810d0-da8c-4f16-a509-a50fe83ae4b2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68\" (UID: \"ec8810d0-da8c-4f16-a509-a50fe83ae4b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68"
Jun 15 06:55:27.922237 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:27.922160    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qqmq\" (UniqueName: \"kubernetes.io/projected/ec8810d0-da8c-4f16-a509-a50fe83ae4b2-kube-api-access-6qqmq\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68\" (UID: \"ec8810d0-da8c-4f16-a509-a50fe83ae4b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68"
Jun 15 06:55:27.922608 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:27.922584    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec8810d0-da8c-4f16-a509-a50fe83ae4b2-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68\" (UID: \"ec8810d0-da8c-4f16-a509-a50fe83ae4b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68"
Jun 15 06:55:27.922838 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:27.922820    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec8810d0-da8c-4f16-a509-a50fe83ae4b2-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68\" (UID: \"ec8810d0-da8c-4f16-a509-a50fe83ae4b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68"
Jun 15 06:55:27.924582 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:27.924565    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec8810d0-da8c-4f16-a509-a50fe83ae4b2-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68\" (UID: \"ec8810d0-da8c-4f16-a509-a50fe83ae4b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68"
Jun 15 06:55:27.931116 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:27.931092    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qqmq\" (UniqueName: \"kubernetes.io/projected/ec8810d0-da8c-4f16-a509-a50fe83ae4b2-kube-api-access-6qqmq\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68\" (UID: \"ec8810d0-da8c-4f16-a509-a50fe83ae4b2\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68"
Jun 15 06:55:28.089962 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:28.089870    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68"
Jun 15 06:55:28.221694 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:28.221668    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68"]
Jun 15 06:55:28.224184 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:55:28.224159    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec8810d0_da8c_4f16_a509_a50fe83ae4b2.slice/crio-27cb24b961e71c51bdb588f2b3be5f04a0f580ba168201c1c1a30a48fc406fa6 WatchSource:0}: Error finding container 27cb24b961e71c51bdb588f2b3be5f04a0f580ba168201c1c1a30a48fc406fa6: Status 404 returned error can't find the container with id 27cb24b961e71c51bdb588f2b3be5f04a0f580ba168201c1c1a30a48fc406fa6
Jun 15 06:55:28.406862 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:28.406826    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68" event={"ID":"ec8810d0-da8c-4f16-a509-a50fe83ae4b2","Type":"ContainerStarted","Data":"5da62ad551ae5606c45a4e17bce074d912509135aeddd5c35c1abfdf7d14757e"}
Jun 15 06:55:28.407035 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:28.406892    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68" event={"ID":"ec8810d0-da8c-4f16-a509-a50fe83ae4b2","Type":"ContainerStarted","Data":"27cb24b961e71c51bdb588f2b3be5f04a0f580ba168201c1c1a30a48fc406fa6"}
Jun 15 06:55:31.322695 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.322663    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"
Jun 15 06:55:31.352712 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.352677    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97ee3963-34fe-4f7d-8567-4d4a4f63a538-proxy-tls\") pod \"97ee3963-34fe-4f7d-8567-4d4a4f63a538\" (UID: \"97ee3963-34fe-4f7d-8567-4d4a4f63a538\") "
Jun 15 06:55:31.352893 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.352737    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l4z42\" (UniqueName: \"kubernetes.io/projected/97ee3963-34fe-4f7d-8567-4d4a4f63a538-kube-api-access-l4z42\") pod \"97ee3963-34fe-4f7d-8567-4d4a4f63a538\" (UID: \"97ee3963-34fe-4f7d-8567-4d4a4f63a538\") "
Jun 15 06:55:31.352893 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.352776    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97ee3963-34fe-4f7d-8567-4d4a4f63a538-kube-rbac-proxy-sar-config\") pod \"97ee3963-34fe-4f7d-8567-4d4a4f63a538\" (UID: \"97ee3963-34fe-4f7d-8567-4d4a4f63a538\") "
Jun 15 06:55:31.352893 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.352815    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97ee3963-34fe-4f7d-8567-4d4a4f63a538-kserve-provision-location\") pod \"97ee3963-34fe-4f7d-8567-4d4a4f63a538\" (UID: \"97ee3963-34fe-4f7d-8567-4d4a4f63a538\") "
Jun 15 06:55:31.353068 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.352908    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97ee3963-34fe-4f7d-8567-4d4a4f63a538-cabundle-cert\") pod \"97ee3963-34fe-4f7d-8567-4d4a4f63a538\" (UID: \"97ee3963-34fe-4f7d-8567-4d4a4f63a538\") "
Jun 15 06:55:31.353240 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.353145    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ee3963-34fe-4f7d-8567-4d4a4f63a538-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "97ee3963-34fe-4f7d-8567-4d4a4f63a538" (UID: "97ee3963-34fe-4f7d-8567-4d4a4f63a538"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:55:31.353240 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.353207    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97ee3963-34fe-4f7d-8567-4d4a4f63a538-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "97ee3963-34fe-4f7d-8567-4d4a4f63a538" (UID: "97ee3963-34fe-4f7d-8567-4d4a4f63a538"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:55:31.353378 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.353312    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97ee3963-34fe-4f7d-8567-4d4a4f63a538-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "97ee3963-34fe-4f7d-8567-4d4a4f63a538" (UID: "97ee3963-34fe-4f7d-8567-4d4a4f63a538"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:55:31.354883 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.354857    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97ee3963-34fe-4f7d-8567-4d4a4f63a538-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "97ee3963-34fe-4f7d-8567-4d4a4f63a538" (UID: "97ee3963-34fe-4f7d-8567-4d4a4f63a538"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:55:31.354978 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.354905    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97ee3963-34fe-4f7d-8567-4d4a4f63a538-kube-api-access-l4z42" (OuterVolumeSpecName: "kube-api-access-l4z42") pod "97ee3963-34fe-4f7d-8567-4d4a4f63a538" (UID: "97ee3963-34fe-4f7d-8567-4d4a4f63a538"). InnerVolumeSpecName "kube-api-access-l4z42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:55:31.422759 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.422726    2572 generic.go:356] "Generic (PLEG): container finished" podID="97ee3963-34fe-4f7d-8567-4d4a4f63a538" containerID="b95bbe7efc2b80f4dfa186e54575854600ff9489215fe3cc0c971d47c95c56f7" exitCode=0
Jun 15 06:55:31.422969 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.422786    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"
Jun 15 06:55:31.422969 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.422791    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v" event={"ID":"97ee3963-34fe-4f7d-8567-4d4a4f63a538","Type":"ContainerDied","Data":"b95bbe7efc2b80f4dfa186e54575854600ff9489215fe3cc0c971d47c95c56f7"}
Jun 15 06:55:31.422969 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.422854    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v" event={"ID":"97ee3963-34fe-4f7d-8567-4d4a4f63a538","Type":"ContainerDied","Data":"f4aceaf607b7f35b81bacddfb30561b34df06697d1944bb9502cdf101af39097"}
Jun 15 06:55:31.422969 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.422881    2572 scope.go:117] "RemoveContainer" containerID="13f1a3f378f010be54c3b9ac66572089f7d494340e64cadefc54d90961bbddef"
Jun 15 06:55:31.433060 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.433037    2572 scope.go:117] "RemoveContainer" containerID="b95bbe7efc2b80f4dfa186e54575854600ff9489215fe3cc0c971d47c95c56f7"
Jun 15 06:55:31.442282 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.442264    2572 scope.go:117] "RemoveContainer" containerID="f5996bf2624966bea89558c989e0ed795615d7f3f55d022b7d8e6b3e780424a8"
Jun 15 06:55:31.451475 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.451433    2572 scope.go:117] "RemoveContainer" containerID="13f1a3f378f010be54c3b9ac66572089f7d494340e64cadefc54d90961bbddef"
Jun 15 06:55:31.452152 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:55:31.452121    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13f1a3f378f010be54c3b9ac66572089f7d494340e64cadefc54d90961bbddef\": container with ID starting with 13f1a3f378f010be54c3b9ac66572089f7d494340e64cadefc54d90961bbddef not found: ID does not exist" containerID="13f1a3f378f010be54c3b9ac66572089f7d494340e64cadefc54d90961bbddef"
Jun 15 06:55:31.452302 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.452162    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13f1a3f378f010be54c3b9ac66572089f7d494340e64cadefc54d90961bbddef"} err="failed to get container status \"13f1a3f378f010be54c3b9ac66572089f7d494340e64cadefc54d90961bbddef\": rpc error: code = NotFound desc = could not find container \"13f1a3f378f010be54c3b9ac66572089f7d494340e64cadefc54d90961bbddef\": container with ID starting with 13f1a3f378f010be54c3b9ac66572089f7d494340e64cadefc54d90961bbddef not found: ID does not exist"
Jun 15 06:55:31.452302 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.452187    2572 scope.go:117] "RemoveContainer" containerID="b95bbe7efc2b80f4dfa186e54575854600ff9489215fe3cc0c971d47c95c56f7"
Jun 15 06:55:31.452523 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:55:31.452503    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b95bbe7efc2b80f4dfa186e54575854600ff9489215fe3cc0c971d47c95c56f7\": container with ID starting with b95bbe7efc2b80f4dfa186e54575854600ff9489215fe3cc0c971d47c95c56f7 not found: ID does not exist" containerID="b95bbe7efc2b80f4dfa186e54575854600ff9489215fe3cc0c971d47c95c56f7"
Jun 15 06:55:31.452603 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.452529    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b95bbe7efc2b80f4dfa186e54575854600ff9489215fe3cc0c971d47c95c56f7"} err="failed to get container status \"b95bbe7efc2b80f4dfa186e54575854600ff9489215fe3cc0c971d47c95c56f7\": rpc error: code = NotFound desc = could not find container \"b95bbe7efc2b80f4dfa186e54575854600ff9489215fe3cc0c971d47c95c56f7\": container with ID starting with b95bbe7efc2b80f4dfa186e54575854600ff9489215fe3cc0c971d47c95c56f7 not found: ID does not exist"
Jun 15 06:55:31.452603 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.452544    2572 scope.go:117] "RemoveContainer" containerID="f5996bf2624966bea89558c989e0ed795615d7f3f55d022b7d8e6b3e780424a8"
Jun 15 06:55:31.452828 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:55:31.452805    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5996bf2624966bea89558c989e0ed795615d7f3f55d022b7d8e6b3e780424a8\": container with ID starting with f5996bf2624966bea89558c989e0ed795615d7f3f55d022b7d8e6b3e780424a8 not found: ID does not exist" containerID="f5996bf2624966bea89558c989e0ed795615d7f3f55d022b7d8e6b3e780424a8"
Jun 15 06:55:31.452889 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.452834    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5996bf2624966bea89558c989e0ed795615d7f3f55d022b7d8e6b3e780424a8"} err="failed to get container status \"f5996bf2624966bea89558c989e0ed795615d7f3f55d022b7d8e6b3e780424a8\": rpc error: code = NotFound desc = could not find container \"f5996bf2624966bea89558c989e0ed795615d7f3f55d022b7d8e6b3e780424a8\": container with ID starting with f5996bf2624966bea89558c989e0ed795615d7f3f55d022b7d8e6b3e780424a8 not found: ID does not exist"
Jun 15 06:55:31.454129 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.454099    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"]
Jun 15 06:55:31.454271 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.454255    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/97ee3963-34fe-4f7d-8567-4d4a4f63a538-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:55:31.454327 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.454278    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97ee3963-34fe-4f7d-8567-4d4a4f63a538-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:55:31.454327 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.454290    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4z42\" (UniqueName: \"kubernetes.io/projected/97ee3963-34fe-4f7d-8567-4d4a4f63a538-kube-api-access-l4z42\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:55:31.454327 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.454299    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97ee3963-34fe-4f7d-8567-4d4a4f63a538-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:55:31.454327 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.454308    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97ee3963-34fe-4f7d-8567-4d4a4f63a538-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:55:31.463350 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:31.463326    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-688f86646d-vcr9v"]
Jun 15 06:55:32.428817 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:32.428724    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68_ec8810d0-da8c-4f16-a509-a50fe83ae4b2/storage-initializer/0.log"
Jun 15 06:55:32.428817 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:32.428780    2572 generic.go:356] "Generic (PLEG): container finished" podID="ec8810d0-da8c-4f16-a509-a50fe83ae4b2" containerID="5da62ad551ae5606c45a4e17bce074d912509135aeddd5c35c1abfdf7d14757e" exitCode=1
Jun 15 06:55:32.429298 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:32.428858    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68" event={"ID":"ec8810d0-da8c-4f16-a509-a50fe83ae4b2","Type":"ContainerDied","Data":"5da62ad551ae5606c45a4e17bce074d912509135aeddd5c35c1abfdf7d14757e"}
Jun 15 06:55:33.324234 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:33.324200    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97ee3963-34fe-4f7d-8567-4d4a4f63a538" path="/var/lib/kubelet/pods/97ee3963-34fe-4f7d-8567-4d4a4f63a538/volumes"
Jun 15 06:55:33.435878 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:33.435852    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68_ec8810d0-da8c-4f16-a509-a50fe83ae4b2/storage-initializer/0.log"
Jun 15 06:55:33.436269 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:33.435972    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68" event={"ID":"ec8810d0-da8c-4f16-a509-a50fe83ae4b2","Type":"ContainerStarted","Data":"3266d364fb52a7f03cfe875b1841be0acb93fd248092efb7831cf4e490b4a012"}
Jun 15 06:55:37.752226 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:37.752196    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68"]
Jun 15 06:55:37.754587 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:37.752414    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68" podUID="ec8810d0-da8c-4f16-a509-a50fe83ae4b2" containerName="storage-initializer" containerID="cri-o://3266d364fb52a7f03cfe875b1841be0acb93fd248092efb7831cf4e490b4a012" gracePeriod=30
Jun 15 06:55:38.463890 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.463864    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68_ec8810d0-da8c-4f16-a509-a50fe83ae4b2/storage-initializer/1.log"
Jun 15 06:55:38.464302 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.464287    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68_ec8810d0-da8c-4f16-a509-a50fe83ae4b2/storage-initializer/0.log"
Jun 15 06:55:38.464372 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.464323    2572 generic.go:356] "Generic (PLEG): container finished" podID="ec8810d0-da8c-4f16-a509-a50fe83ae4b2" containerID="3266d364fb52a7f03cfe875b1841be0acb93fd248092efb7831cf4e490b4a012" exitCode=1
Jun 15 06:55:38.464414 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.464393    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68" event={"ID":"ec8810d0-da8c-4f16-a509-a50fe83ae4b2","Type":"ContainerDied","Data":"3266d364fb52a7f03cfe875b1841be0acb93fd248092efb7831cf4e490b4a012"}
Jun 15 06:55:38.464478 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.464468    2572 scope.go:117] "RemoveContainer" containerID="5da62ad551ae5606c45a4e17bce074d912509135aeddd5c35c1abfdf7d14757e"
Jun 15 06:55:38.492676 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.492657    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68_ec8810d0-da8c-4f16-a509-a50fe83ae4b2/storage-initializer/1.log"
Jun 15 06:55:38.496131 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.496116    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68"
Jun 15 06:55:38.629902 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.629814    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec8810d0-da8c-4f16-a509-a50fe83ae4b2-kserve-provision-location\") pod \"ec8810d0-da8c-4f16-a509-a50fe83ae4b2\" (UID: \"ec8810d0-da8c-4f16-a509-a50fe83ae4b2\") "
Jun 15 06:55:38.629902 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.629849    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec8810d0-da8c-4f16-a509-a50fe83ae4b2-proxy-tls\") pod \"ec8810d0-da8c-4f16-a509-a50fe83ae4b2\" (UID: \"ec8810d0-da8c-4f16-a509-a50fe83ae4b2\") "
Jun 15 06:55:38.629902 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.629902    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qqmq\" (UniqueName: \"kubernetes.io/projected/ec8810d0-da8c-4f16-a509-a50fe83ae4b2-kube-api-access-6qqmq\") pod \"ec8810d0-da8c-4f16-a509-a50fe83ae4b2\" (UID: \"ec8810d0-da8c-4f16-a509-a50fe83ae4b2\") "
Jun 15 06:55:38.630203 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.629932    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec8810d0-da8c-4f16-a509-a50fe83ae4b2-kube-rbac-proxy-sar-config\") pod \"ec8810d0-da8c-4f16-a509-a50fe83ae4b2\" (UID: \"ec8810d0-da8c-4f16-a509-a50fe83ae4b2\") "
Jun 15 06:55:38.630203 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.630145    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec8810d0-da8c-4f16-a509-a50fe83ae4b2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ec8810d0-da8c-4f16-a509-a50fe83ae4b2" (UID: "ec8810d0-da8c-4f16-a509-a50fe83ae4b2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:55:38.630386 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.630343    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec8810d0-da8c-4f16-a509-a50fe83ae4b2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ec8810d0-da8c-4f16-a509-a50fe83ae4b2" (UID: "ec8810d0-da8c-4f16-a509-a50fe83ae4b2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:55:38.632011 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.631990    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec8810d0-da8c-4f16-a509-a50fe83ae4b2-kube-api-access-6qqmq" (OuterVolumeSpecName: "kube-api-access-6qqmq") pod "ec8810d0-da8c-4f16-a509-a50fe83ae4b2" (UID: "ec8810d0-da8c-4f16-a509-a50fe83ae4b2"). InnerVolumeSpecName "kube-api-access-6qqmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:55:38.632073 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.632020    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec8810d0-da8c-4f16-a509-a50fe83ae4b2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ec8810d0-da8c-4f16-a509-a50fe83ae4b2" (UID: "ec8810d0-da8c-4f16-a509-a50fe83ae4b2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:55:38.730732 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.730694    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ec8810d0-da8c-4f16-a509-a50fe83ae4b2-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:55:38.730732 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.730725    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ec8810d0-da8c-4f16-a509-a50fe83ae4b2-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:55:38.730929 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.730740    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qqmq\" (UniqueName: \"kubernetes.io/projected/ec8810d0-da8c-4f16-a509-a50fe83ae4b2-kube-api-access-6qqmq\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:55:38.730929 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.730753    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ec8810d0-da8c-4f16-a509-a50fe83ae4b2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:55:38.820124 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.820091    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78"]
Jun 15 06:55:38.820501 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.820481    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec8810d0-da8c-4f16-a509-a50fe83ae4b2" containerName="storage-initializer"
Jun 15 06:55:38.820501 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.820495    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec8810d0-da8c-4f16-a509-a50fe83ae4b2" containerName="storage-initializer"
Jun 15 06:55:38.820574 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.820508    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97ee3963-34fe-4f7d-8567-4d4a4f63a538" containerName="kube-rbac-proxy"
Jun 15 06:55:38.820574 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.820517    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="97ee3963-34fe-4f7d-8567-4d4a4f63a538" containerName="kube-rbac-proxy"
Jun 15 06:55:38.820574 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.820525    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97ee3963-34fe-4f7d-8567-4d4a4f63a538" containerName="storage-initializer"
Jun 15 06:55:38.820574 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.820530    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="97ee3963-34fe-4f7d-8567-4d4a4f63a538" containerName="storage-initializer"
Jun 15 06:55:38.820574 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.820544    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97ee3963-34fe-4f7d-8567-4d4a4f63a538" containerName="kserve-container"
Jun 15 06:55:38.820574 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.820549    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="97ee3963-34fe-4f7d-8567-4d4a4f63a538" containerName="kserve-container"
Jun 15 06:55:38.820759 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.820602    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="97ee3963-34fe-4f7d-8567-4d4a4f63a538" containerName="kube-rbac-proxy"
Jun 15 06:55:38.820759 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.820614    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec8810d0-da8c-4f16-a509-a50fe83ae4b2" containerName="storage-initializer"
Jun 15 06:55:38.820759 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.820624    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec8810d0-da8c-4f16-a509-a50fe83ae4b2" containerName="storage-initializer"
Jun 15 06:55:38.820759 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.820631    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="97ee3963-34fe-4f7d-8567-4d4a4f63a538" containerName="kserve-container"
Jun 15 06:55:38.820759 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.820694    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec8810d0-da8c-4f16-a509-a50fe83ae4b2" containerName="storage-initializer"
Jun 15 06:55:38.820759 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.820700    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec8810d0-da8c-4f16-a509-a50fe83ae4b2" containerName="storage-initializer"
Jun 15 06:55:38.824009 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.823990    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78"
Jun 15 06:55:38.826048 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.826027    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Jun 15 06:55:38.826048 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.826042    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jun 15 06:55:38.826216 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.826076    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Jun 15 06:55:38.833324 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.833300    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78"]
Jun 15 06:55:38.931789 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.931759    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l26bq\" (UniqueName: \"kubernetes.io/projected/d2f4aa51-b485-41b0-955d-fb91faef78da-kube-api-access-l26bq\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78\" (UID: \"d2f4aa51-b485-41b0-955d-fb91faef78da\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78"
Jun 15 06:55:38.931958 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.931797    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2f4aa51-b485-41b0-955d-fb91faef78da-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78\" (UID: \"d2f4aa51-b485-41b0-955d-fb91faef78da\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78"
Jun 15 06:55:38.931958 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.931891    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2f4aa51-b485-41b0-955d-fb91faef78da-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78\" (UID: \"d2f4aa51-b485-41b0-955d-fb91faef78da\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78"
Jun 15 06:55:38.931958 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.931921    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2f4aa51-b485-41b0-955d-fb91faef78da-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78\" (UID: \"d2f4aa51-b485-41b0-955d-fb91faef78da\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78"
Jun 15 06:55:38.931958 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:38.931945    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2f4aa51-b485-41b0-955d-fb91faef78da-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78\" (UID: \"d2f4aa51-b485-41b0-955d-fb91faef78da\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78"
Jun 15 06:55:39.033205 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:39.033172    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l26bq\" (UniqueName: \"kubernetes.io/projected/d2f4aa51-b485-41b0-955d-fb91faef78da-kube-api-access-l26bq\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78\" (UID: \"d2f4aa51-b485-41b0-955d-fb91faef78da\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78"
Jun 15 06:55:39.033205 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:39.033211    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2f4aa51-b485-41b0-955d-fb91faef78da-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78\" (UID: \"d2f4aa51-b485-41b0-955d-fb91faef78da\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78"
Jun 15 06:55:39.033484 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:39.033273    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2f4aa51-b485-41b0-955d-fb91faef78da-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78\" (UID: \"d2f4aa51-b485-41b0-955d-fb91faef78da\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78"
Jun 15 06:55:39.033484 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:39.033295    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2f4aa51-b485-41b0-955d-fb91faef78da-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78\" (UID: \"d2f4aa51-b485-41b0-955d-fb91faef78da\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78"
Jun 15 06:55:39.033484 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:39.033324    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2f4aa51-b485-41b0-955d-fb91faef78da-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78\" (UID: \"d2f4aa51-b485-41b0-955d-fb91faef78da\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78"
Jun 15 06:55:39.033739 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:39.033716    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2f4aa51-b485-41b0-955d-fb91faef78da-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78\" (UID: \"d2f4aa51-b485-41b0-955d-fb91faef78da\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78"
Jun 15 06:55:39.034130 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:39.034093    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2f4aa51-b485-41b0-955d-fb91faef78da-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78\" (UID: \"d2f4aa51-b485-41b0-955d-fb91faef78da\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78"
Jun 15 06:55:39.034250 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:39.034093    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2f4aa51-b485-41b0-955d-fb91faef78da-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78\" (UID: \"d2f4aa51-b485-41b0-955d-fb91faef78da\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78"
Jun 15 06:55:39.035871 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:39.035849    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2f4aa51-b485-41b0-955d-fb91faef78da-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78\" (UID: \"d2f4aa51-b485-41b0-955d-fb91faef78da\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78"
Jun 15 06:55:39.041940 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:39.041913    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l26bq\" (UniqueName: \"kubernetes.io/projected/d2f4aa51-b485-41b0-955d-fb91faef78da-kube-api-access-l26bq\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78\" (UID: \"d2f4aa51-b485-41b0-955d-fb91faef78da\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78"
Jun 15 06:55:39.135538 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:39.135509    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78"
Jun 15 06:55:39.269573 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:39.269544    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78"]
Jun 15 06:55:39.271322 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:55:39.271297    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2f4aa51_b485_41b0_955d_fb91faef78da.slice/crio-943994efa8ab60bf889d74fdcb114d52483f8a8aef51ea7f38eb04873f4379a0 WatchSource:0}: Error finding container 943994efa8ab60bf889d74fdcb114d52483f8a8aef51ea7f38eb04873f4379a0: Status 404 returned error can't find the container with id 943994efa8ab60bf889d74fdcb114d52483f8a8aef51ea7f38eb04873f4379a0
Jun 15 06:55:39.471537 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:39.471444    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78" event={"ID":"d2f4aa51-b485-41b0-955d-fb91faef78da","Type":"ContainerStarted","Data":"48d4d9647b9d1f522e071fa9b417ef7bb25561edc74822e9a9e9f9b25b582827"}
Jun 15 06:55:39.471537 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:39.471520    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78" event={"ID":"d2f4aa51-b485-41b0-955d-fb91faef78da","Type":"ContainerStarted","Data":"943994efa8ab60bf889d74fdcb114d52483f8a8aef51ea7f38eb04873f4379a0"}
Jun 15 06:55:39.472860 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:39.472838    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68_ec8810d0-da8c-4f16-a509-a50fe83ae4b2/storage-initializer/1.log"
Jun 15 06:55:39.472992 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:39.472921    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68"
Jun 15 06:55:39.472992 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:39.472968    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68" event={"ID":"ec8810d0-da8c-4f16-a509-a50fe83ae4b2","Type":"ContainerDied","Data":"27cb24b961e71c51bdb588f2b3be5f04a0f580ba168201c1c1a30a48fc406fa6"}
Jun 15 06:55:39.473068 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:39.473028    2572 scope.go:117] "RemoveContainer" containerID="3266d364fb52a7f03cfe875b1841be0acb93fd248092efb7831cf4e490b4a012"
Jun 15 06:55:39.563958 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:39.563917    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68"]
Jun 15 06:55:39.567638 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:39.567614    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-bb98d6cd8-6vj68"]
Jun 15 06:55:40.480002 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:40.479967    2572 generic.go:356] "Generic (PLEG): container finished" podID="d2f4aa51-b485-41b0-955d-fb91faef78da" containerID="48d4d9647b9d1f522e071fa9b417ef7bb25561edc74822e9a9e9f9b25b582827" exitCode=0
Jun 15 06:55:40.480425 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:40.480059    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78" event={"ID":"d2f4aa51-b485-41b0-955d-fb91faef78da","Type":"ContainerDied","Data":"48d4d9647b9d1f522e071fa9b417ef7bb25561edc74822e9a9e9f9b25b582827"}
Jun 15 06:55:41.323795 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:41.323764    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec8810d0-da8c-4f16-a509-a50fe83ae4b2" path="/var/lib/kubelet/pods/ec8810d0-da8c-4f16-a509-a50fe83ae4b2/volumes"
Jun 15 06:55:41.488337 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:41.488300    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78" event={"ID":"d2f4aa51-b485-41b0-955d-fb91faef78da","Type":"ContainerStarted","Data":"cf92d5191847f4ca48e543558c58c8fedab7cce09bb8cfafdf318af5fc6de133"}
Jun 15 06:55:41.488799 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:41.488391    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78" event={"ID":"d2f4aa51-b485-41b0-955d-fb91faef78da","Type":"ContainerStarted","Data":"2c68ca47e07a3ea32574a43ba08d540f9d4b8132602d2d28e08239c8b81cc80f"}
Jun 15 06:55:41.488799 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:41.488463    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78"
Jun 15 06:55:41.508945 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:41.508896    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78" podStartSLOduration=2.496180229 podStartE2EDuration="3.508882816s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.012702587s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:55:41.506946313 +0000 UTC m=+4056.655126326" watchObservedRunningTime="2026-06-15 06:55:41.508882816 +0000 UTC m=+4056.657062885"
Jun 15 06:55:42.495559 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:42.495531    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78"
Jun 15 06:55:42.497073 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:42.497045    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78" podUID="d2f4aa51-b485-41b0-955d-fb91faef78da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Jun 15 06:55:43.500569 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:43.500528    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78" podUID="d2f4aa51-b485-41b0-955d-fb91faef78da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Jun 15 06:55:48.505747 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:48.505719    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78"
Jun 15 06:55:48.506231 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:48.506204    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78" podUID="d2f4aa51-b485-41b0-955d-fb91faef78da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Jun 15 06:55:58.506409 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:55:58.506349    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78" podUID="d2f4aa51-b485-41b0-955d-fb91faef78da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Jun 15 06:56:08.506313 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:56:08.506274    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78" podUID="d2f4aa51-b485-41b0-955d-fb91faef78da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Jun 15 06:56:18.506510 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:56:18.506467    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78" podUID="d2f4aa51-b485-41b0-955d-fb91faef78da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Jun 15 06:56:28.506470 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:56:28.506431    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78" podUID="d2f4aa51-b485-41b0-955d-fb91faef78da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Jun 15 06:56:38.506506 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:56:38.506461    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78" podUID="d2f4aa51-b485-41b0-955d-fb91faef78da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Jun 15 06:56:48.506573 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:56:48.506488    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78" podUID="d2f4aa51-b485-41b0-955d-fb91faef78da" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.64:8080: connect: connection refused"
Jun 15 06:56:58.506994 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:56:58.506965    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78"
Jun 15 06:56:58.861875 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:56:58.861790    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78"]
Jun 15 06:56:58.862069 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:56:58.862021    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78" podUID="d2f4aa51-b485-41b0-955d-fb91faef78da" containerName="kserve-container" containerID="cri-o://2c68ca47e07a3ea32574a43ba08d540f9d4b8132602d2d28e08239c8b81cc80f" gracePeriod=30
Jun 15 06:56:58.862069 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:56:58.862048    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78" podUID="d2f4aa51-b485-41b0-955d-fb91faef78da" containerName="kube-rbac-proxy" containerID="cri-o://cf92d5191847f4ca48e543558c58c8fedab7cce09bb8cfafdf318af5fc6de133" gracePeriod=30
Jun 15 06:56:59.865712 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:56:59.865681    2572 generic.go:356] "Generic (PLEG): container finished" podID="d2f4aa51-b485-41b0-955d-fb91faef78da" containerID="cf92d5191847f4ca48e543558c58c8fedab7cce09bb8cfafdf318af5fc6de133" exitCode=2
Jun 15 06:56:59.866157 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:56:59.865762    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78" event={"ID":"d2f4aa51-b485-41b0-955d-fb91faef78da","Type":"ContainerDied","Data":"cf92d5191847f4ca48e543558c58c8fedab7cce09bb8cfafdf318af5fc6de133"}
Jun 15 06:56:59.901700 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:56:59.901665    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh"]
Jun 15 06:56:59.905563 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:56:59.905543    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh"
Jun 15 06:56:59.907515 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:56:59.907496    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Jun 15 06:56:59.907631 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:56:59.907612    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Jun 15 06:56:59.915623 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:56:59.915602    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh"]
Jun 15 06:56:59.980070 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:56:59.980039    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1a24661-6e05-4139-8b73-f54fa2fac3a1-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh\" (UID: \"f1a24661-6e05-4139-8b73-f54fa2fac3a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh"
Jun 15 06:56:59.980224 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:56:59.980089    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1a24661-6e05-4139-8b73-f54fa2fac3a1-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh\" (UID: \"f1a24661-6e05-4139-8b73-f54fa2fac3a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh"
Jun 15 06:56:59.980224 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:56:59.980108    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1a24661-6e05-4139-8b73-f54fa2fac3a1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh\" (UID: \"f1a24661-6e05-4139-8b73-f54fa2fac3a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh"
Jun 15 06:56:59.980224 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:56:59.980146    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4tm8\" (UniqueName: \"kubernetes.io/projected/f1a24661-6e05-4139-8b73-f54fa2fac3a1-kube-api-access-h4tm8\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh\" (UID: \"f1a24661-6e05-4139-8b73-f54fa2fac3a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh"
Jun 15 06:57:00.081273 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:00.081239    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1a24661-6e05-4139-8b73-f54fa2fac3a1-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh\" (UID: \"f1a24661-6e05-4139-8b73-f54fa2fac3a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh"
Jun 15 06:57:00.081475 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:00.081303    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1a24661-6e05-4139-8b73-f54fa2fac3a1-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh\" (UID: \"f1a24661-6e05-4139-8b73-f54fa2fac3a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh"
Jun 15 06:57:00.081475 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:00.081324    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1a24661-6e05-4139-8b73-f54fa2fac3a1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh\" (UID: \"f1a24661-6e05-4139-8b73-f54fa2fac3a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh"
Jun 15 06:57:00.081475 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:57:00.081347    2572 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jun 15 06:57:00.081475 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:57:00.081433    2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1a24661-6e05-4139-8b73-f54fa2fac3a1-proxy-tls podName:f1a24661-6e05-4139-8b73-f54fa2fac3a1 nodeName:}" failed. No retries permitted until 2026-06-15 06:57:00.58141469 +0000 UTC m=+4135.729594666 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f1a24661-6e05-4139-8b73-f54fa2fac3a1-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh" (UID: "f1a24661-6e05-4139-8b73-f54fa2fac3a1") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jun 15 06:57:00.081475 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:00.081349    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4tm8\" (UniqueName: \"kubernetes.io/projected/f1a24661-6e05-4139-8b73-f54fa2fac3a1-kube-api-access-h4tm8\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh\" (UID: \"f1a24661-6e05-4139-8b73-f54fa2fac3a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh"
Jun 15 06:57:00.081773 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:00.081752    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1a24661-6e05-4139-8b73-f54fa2fac3a1-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh\" (UID: \"f1a24661-6e05-4139-8b73-f54fa2fac3a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh"
Jun 15 06:57:00.082015 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:00.081998    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1a24661-6e05-4139-8b73-f54fa2fac3a1-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh\" (UID: \"f1a24661-6e05-4139-8b73-f54fa2fac3a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh"
Jun 15 06:57:00.092059 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:00.092027    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4tm8\" (UniqueName: \"kubernetes.io/projected/f1a24661-6e05-4139-8b73-f54fa2fac3a1-kube-api-access-h4tm8\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh\" (UID: \"f1a24661-6e05-4139-8b73-f54fa2fac3a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh"
Jun 15 06:57:00.585664 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:00.585626    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1a24661-6e05-4139-8b73-f54fa2fac3a1-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh\" (UID: \"f1a24661-6e05-4139-8b73-f54fa2fac3a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh"
Jun 15 06:57:00.588160 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:00.588133    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1a24661-6e05-4139-8b73-f54fa2fac3a1-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh\" (UID: \"f1a24661-6e05-4139-8b73-f54fa2fac3a1\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh"
Jun 15 06:57:00.816020 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:00.815985    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh"
Jun 15 06:57:00.953756 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:00.953730    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh"]
Jun 15 06:57:00.956315 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:57:00.956293    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1a24661_6e05_4139_8b73_f54fa2fac3a1.slice/crio-716a92ef1f21ee5b144ac435c612d655986af08e68fa3a8c15386a351aa53e09 WatchSource:0}: Error finding container 716a92ef1f21ee5b144ac435c612d655986af08e68fa3a8c15386a351aa53e09: Status 404 returned error can't find the container with id 716a92ef1f21ee5b144ac435c612d655986af08e68fa3a8c15386a351aa53e09
Jun 15 06:57:01.880876 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:01.880843    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh" event={"ID":"f1a24661-6e05-4139-8b73-f54fa2fac3a1","Type":"ContainerStarted","Data":"9ef125fa1650bb72b17bc14393ac9f35207520f68dcc61a2790ec2006c331bfc"}
Jun 15 06:57:01.881045 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:01.880904    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh" event={"ID":"f1a24661-6e05-4139-8b73-f54fa2fac3a1","Type":"ContainerStarted","Data":"716a92ef1f21ee5b144ac435c612d655986af08e68fa3a8c15386a351aa53e09"}
Jun 15 06:57:03.501684 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.501635    2572 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78" podUID="d2f4aa51-b485-41b0-955d-fb91faef78da" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.64:8643/healthz\": dial tcp 10.134.0.64:8643: connect: connection refused"
Jun 15 06:57:03.709075 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.709050    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78"
Jun 15 06:57:03.813855 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.813753    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2f4aa51-b485-41b0-955d-fb91faef78da-kserve-provision-location\") pod \"d2f4aa51-b485-41b0-955d-fb91faef78da\" (UID: \"d2f4aa51-b485-41b0-955d-fb91faef78da\") "
Jun 15 06:57:03.813855 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.813839    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l26bq\" (UniqueName: \"kubernetes.io/projected/d2f4aa51-b485-41b0-955d-fb91faef78da-kube-api-access-l26bq\") pod \"d2f4aa51-b485-41b0-955d-fb91faef78da\" (UID: \"d2f4aa51-b485-41b0-955d-fb91faef78da\") "
Jun 15 06:57:03.813855 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.813857    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2f4aa51-b485-41b0-955d-fb91faef78da-kube-rbac-proxy-sar-config\") pod \"d2f4aa51-b485-41b0-955d-fb91faef78da\" (UID: \"d2f4aa51-b485-41b0-955d-fb91faef78da\") "
Jun 15 06:57:03.814139 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.813875    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2f4aa51-b485-41b0-955d-fb91faef78da-proxy-tls\") pod \"d2f4aa51-b485-41b0-955d-fb91faef78da\" (UID: \"d2f4aa51-b485-41b0-955d-fb91faef78da\") "
Jun 15 06:57:03.814139 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.813898    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2f4aa51-b485-41b0-955d-fb91faef78da-cabundle-cert\") pod \"d2f4aa51-b485-41b0-955d-fb91faef78da\" (UID: \"d2f4aa51-b485-41b0-955d-fb91faef78da\") "
Jun 15 06:57:03.814258 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.814148    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f4aa51-b485-41b0-955d-fb91faef78da-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d2f4aa51-b485-41b0-955d-fb91faef78da" (UID: "d2f4aa51-b485-41b0-955d-fb91faef78da"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:57:03.814258 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.814195    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f4aa51-b485-41b0-955d-fb91faef78da-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d2f4aa51-b485-41b0-955d-fb91faef78da" (UID: "d2f4aa51-b485-41b0-955d-fb91faef78da"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:57:03.814426 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.814404    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2f4aa51-b485-41b0-955d-fb91faef78da-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d2f4aa51-b485-41b0-955d-fb91faef78da" (UID: "d2f4aa51-b485-41b0-955d-fb91faef78da"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:57:03.815998 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.815975    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2f4aa51-b485-41b0-955d-fb91faef78da-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d2f4aa51-b485-41b0-955d-fb91faef78da" (UID: "d2f4aa51-b485-41b0-955d-fb91faef78da"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:57:03.816119 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.816099    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f4aa51-b485-41b0-955d-fb91faef78da-kube-api-access-l26bq" (OuterVolumeSpecName: "kube-api-access-l26bq") pod "d2f4aa51-b485-41b0-955d-fb91faef78da" (UID: "d2f4aa51-b485-41b0-955d-fb91faef78da"). InnerVolumeSpecName "kube-api-access-l26bq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:57:03.891094 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.891062    2572 generic.go:356] "Generic (PLEG): container finished" podID="d2f4aa51-b485-41b0-955d-fb91faef78da" containerID="2c68ca47e07a3ea32574a43ba08d540f9d4b8132602d2d28e08239c8b81cc80f" exitCode=0
Jun 15 06:57:03.891269 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.891124    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78"
Jun 15 06:57:03.891269 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.891136    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78" event={"ID":"d2f4aa51-b485-41b0-955d-fb91faef78da","Type":"ContainerDied","Data":"2c68ca47e07a3ea32574a43ba08d540f9d4b8132602d2d28e08239c8b81cc80f"}
Jun 15 06:57:03.891269 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.891209    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78" event={"ID":"d2f4aa51-b485-41b0-955d-fb91faef78da","Type":"ContainerDied","Data":"943994efa8ab60bf889d74fdcb114d52483f8a8aef51ea7f38eb04873f4379a0"}
Jun 15 06:57:03.891269 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.891248    2572 scope.go:117] "RemoveContainer" containerID="cf92d5191847f4ca48e543558c58c8fedab7cce09bb8cfafdf318af5fc6de133"
Jun 15 06:57:03.900423 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.900405    2572 scope.go:117] "RemoveContainer" containerID="2c68ca47e07a3ea32574a43ba08d540f9d4b8132602d2d28e08239c8b81cc80f"
Jun 15 06:57:03.908858 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.908839    2572 scope.go:117] "RemoveContainer" containerID="48d4d9647b9d1f522e071fa9b417ef7bb25561edc74822e9a9e9f9b25b582827"
Jun 15 06:57:03.915077 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.915058    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2f4aa51-b485-41b0-955d-fb91faef78da-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:57:03.915154 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.915079    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l26bq\" (UniqueName: \"kubernetes.io/projected/d2f4aa51-b485-41b0-955d-fb91faef78da-kube-api-access-l26bq\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:57:03.915154 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.915089    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d2f4aa51-b485-41b0-955d-fb91faef78da-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:57:03.915154 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.915099    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d2f4aa51-b485-41b0-955d-fb91faef78da-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:57:03.915154 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.915109    2572 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d2f4aa51-b485-41b0-955d-fb91faef78da-cabundle-cert\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:57:03.915870 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.915858    2572 scope.go:117] "RemoveContainer" containerID="cf92d5191847f4ca48e543558c58c8fedab7cce09bb8cfafdf318af5fc6de133"
Jun 15 06:57:03.916116 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:57:03.916096    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf92d5191847f4ca48e543558c58c8fedab7cce09bb8cfafdf318af5fc6de133\": container with ID starting with cf92d5191847f4ca48e543558c58c8fedab7cce09bb8cfafdf318af5fc6de133 not found: ID does not exist" containerID="cf92d5191847f4ca48e543558c58c8fedab7cce09bb8cfafdf318af5fc6de133"
Jun 15 06:57:03.916155 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.916129    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf92d5191847f4ca48e543558c58c8fedab7cce09bb8cfafdf318af5fc6de133"} err="failed to get container status \"cf92d5191847f4ca48e543558c58c8fedab7cce09bb8cfafdf318af5fc6de133\": rpc error: code = NotFound desc = could not find container \"cf92d5191847f4ca48e543558c58c8fedab7cce09bb8cfafdf318af5fc6de133\": container with ID starting with cf92d5191847f4ca48e543558c58c8fedab7cce09bb8cfafdf318af5fc6de133 not found: ID does not exist"
Jun 15 06:57:03.916155 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.916147    2572 scope.go:117] "RemoveContainer" containerID="2c68ca47e07a3ea32574a43ba08d540f9d4b8132602d2d28e08239c8b81cc80f"
Jun 15 06:57:03.916412 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:57:03.916393    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c68ca47e07a3ea32574a43ba08d540f9d4b8132602d2d28e08239c8b81cc80f\": container with ID starting with 2c68ca47e07a3ea32574a43ba08d540f9d4b8132602d2d28e08239c8b81cc80f not found: ID does not exist" containerID="2c68ca47e07a3ea32574a43ba08d540f9d4b8132602d2d28e08239c8b81cc80f"
Jun 15 06:57:03.916455 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.916418    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c68ca47e07a3ea32574a43ba08d540f9d4b8132602d2d28e08239c8b81cc80f"} err="failed to get container status \"2c68ca47e07a3ea32574a43ba08d540f9d4b8132602d2d28e08239c8b81cc80f\": rpc error: code = NotFound desc = could not find container \"2c68ca47e07a3ea32574a43ba08d540f9d4b8132602d2d28e08239c8b81cc80f\": container with ID starting with 2c68ca47e07a3ea32574a43ba08d540f9d4b8132602d2d28e08239c8b81cc80f not found: ID does not exist"
Jun 15 06:57:03.916455 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.916432    2572 scope.go:117] "RemoveContainer" containerID="48d4d9647b9d1f522e071fa9b417ef7bb25561edc74822e9a9e9f9b25b582827"
Jun 15 06:57:03.916639 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:57:03.916621    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48d4d9647b9d1f522e071fa9b417ef7bb25561edc74822e9a9e9f9b25b582827\": container with ID starting with 48d4d9647b9d1f522e071fa9b417ef7bb25561edc74822e9a9e9f9b25b582827 not found: ID does not exist" containerID="48d4d9647b9d1f522e071fa9b417ef7bb25561edc74822e9a9e9f9b25b582827"
Jun 15 06:57:03.916679 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.916644    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48d4d9647b9d1f522e071fa9b417ef7bb25561edc74822e9a9e9f9b25b582827"} err="failed to get container status \"48d4d9647b9d1f522e071fa9b417ef7bb25561edc74822e9a9e9f9b25b582827\": rpc error: code = NotFound desc = could not find container \"48d4d9647b9d1f522e071fa9b417ef7bb25561edc74822e9a9e9f9b25b582827\": container with ID starting with 48d4d9647b9d1f522e071fa9b417ef7bb25561edc74822e9a9e9f9b25b582827 not found: ID does not exist"
Jun 15 06:57:03.925242 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.925220    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78"]
Jun 15 06:57:03.940944 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:03.940922    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-c6db6db59-xjv78"]
Jun 15 06:57:05.324275 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:05.324192    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2f4aa51-b485-41b0-955d-fb91faef78da" path="/var/lib/kubelet/pods/d2f4aa51-b485-41b0-955d-fb91faef78da/volumes"
Jun 15 06:57:05.903732 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:05.903708    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh_f1a24661-6e05-4139-8b73-f54fa2fac3a1/storage-initializer/0.log"
Jun 15 06:57:05.903860 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:05.903750    2572 generic.go:356] "Generic (PLEG): container finished" podID="f1a24661-6e05-4139-8b73-f54fa2fac3a1" containerID="9ef125fa1650bb72b17bc14393ac9f35207520f68dcc61a2790ec2006c331bfc" exitCode=1
Jun 15 06:57:05.903860 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:05.903799    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh" event={"ID":"f1a24661-6e05-4139-8b73-f54fa2fac3a1","Type":"ContainerDied","Data":"9ef125fa1650bb72b17bc14393ac9f35207520f68dcc61a2790ec2006c331bfc"}
Jun 15 06:57:06.910155 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:06.910130    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh_f1a24661-6e05-4139-8b73-f54fa2fac3a1/storage-initializer/0.log"
Jun 15 06:57:06.910548 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:06.910246    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh" event={"ID":"f1a24661-6e05-4139-8b73-f54fa2fac3a1","Type":"ContainerStarted","Data":"b24ab4fd9af791b08c2f088339af14a5c745a65f4e6458190233235e5f2f01b3"}
Jun 15 06:57:09.912307 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:09.912277    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh"]
Jun 15 06:57:09.912740 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:09.912485    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh" podUID="f1a24661-6e05-4139-8b73-f54fa2fac3a1" containerName="storage-initializer" containerID="cri-o://b24ab4fd9af791b08c2f088339af14a5c745a65f4e6458190233235e5f2f01b3" gracePeriod=30
Jun 15 06:57:09.925183 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:09.925155    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh_f1a24661-6e05-4139-8b73-f54fa2fac3a1/storage-initializer/1.log"
Jun 15 06:57:09.925623 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:09.925604    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh_f1a24661-6e05-4139-8b73-f54fa2fac3a1/storage-initializer/0.log"
Jun 15 06:57:09.925744 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:09.925655    2572 generic.go:356] "Generic (PLEG): container finished" podID="f1a24661-6e05-4139-8b73-f54fa2fac3a1" containerID="b24ab4fd9af791b08c2f088339af14a5c745a65f4e6458190233235e5f2f01b3" exitCode=1
Jun 15 06:57:09.925744 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:09.925687    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh" event={"ID":"f1a24661-6e05-4139-8b73-f54fa2fac3a1","Type":"ContainerDied","Data":"b24ab4fd9af791b08c2f088339af14a5c745a65f4e6458190233235e5f2f01b3"}
Jun 15 06:57:09.925894 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:09.925756    2572 scope.go:117] "RemoveContainer" containerID="9ef125fa1650bb72b17bc14393ac9f35207520f68dcc61a2790ec2006c331bfc"
Jun 15 06:57:10.048108 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:10.048086    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh_f1a24661-6e05-4139-8b73-f54fa2fac3a1/storage-initializer/1.log"
Jun 15 06:57:10.051588 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:10.051571    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh"
Jun 15 06:57:10.173168 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:10.173079    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1a24661-6e05-4139-8b73-f54fa2fac3a1-kserve-provision-location\") pod \"f1a24661-6e05-4139-8b73-f54fa2fac3a1\" (UID: \"f1a24661-6e05-4139-8b73-f54fa2fac3a1\") "
Jun 15 06:57:10.173168 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:10.173167    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1a24661-6e05-4139-8b73-f54fa2fac3a1-kube-rbac-proxy-sar-config\") pod \"f1a24661-6e05-4139-8b73-f54fa2fac3a1\" (UID: \"f1a24661-6e05-4139-8b73-f54fa2fac3a1\") "
Jun 15 06:57:10.173351 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:10.173256    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1a24661-6e05-4139-8b73-f54fa2fac3a1-proxy-tls\") pod \"f1a24661-6e05-4139-8b73-f54fa2fac3a1\" (UID: \"f1a24661-6e05-4139-8b73-f54fa2fac3a1\") "
Jun 15 06:57:10.173351 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:10.173284    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4tm8\" (UniqueName: \"kubernetes.io/projected/f1a24661-6e05-4139-8b73-f54fa2fac3a1-kube-api-access-h4tm8\") pod \"f1a24661-6e05-4139-8b73-f54fa2fac3a1\" (UID: \"f1a24661-6e05-4139-8b73-f54fa2fac3a1\") "
Jun 15 06:57:10.173464 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:10.173446    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1a24661-6e05-4139-8b73-f54fa2fac3a1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f1a24661-6e05-4139-8b73-f54fa2fac3a1" (UID: "f1a24661-6e05-4139-8b73-f54fa2fac3a1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:57:10.173577 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:10.173563    2572 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f1a24661-6e05-4139-8b73-f54fa2fac3a1-kserve-provision-location\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:57:10.173621 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:10.173576    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1a24661-6e05-4139-8b73-f54fa2fac3a1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f1a24661-6e05-4139-8b73-f54fa2fac3a1" (UID: "f1a24661-6e05-4139-8b73-f54fa2fac3a1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jun 15 06:57:10.175393 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:10.175352    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1a24661-6e05-4139-8b73-f54fa2fac3a1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f1a24661-6e05-4139-8b73-f54fa2fac3a1" (UID: "f1a24661-6e05-4139-8b73-f54fa2fac3a1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jun 15 06:57:10.175466 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:10.175375    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a24661-6e05-4139-8b73-f54fa2fac3a1-kube-api-access-h4tm8" (OuterVolumeSpecName: "kube-api-access-h4tm8") pod "f1a24661-6e05-4139-8b73-f54fa2fac3a1" (UID: "f1a24661-6e05-4139-8b73-f54fa2fac3a1"). InnerVolumeSpecName "kube-api-access-h4tm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:57:10.274662 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:10.274624    2572 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f1a24661-6e05-4139-8b73-f54fa2fac3a1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:57:10.274662 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:10.274653    2572 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f1a24661-6e05-4139-8b73-f54fa2fac3a1-proxy-tls\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:57:10.274662 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:10.274663    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4tm8\" (UniqueName: \"kubernetes.io/projected/f1a24661-6e05-4139-8b73-f54fa2fac3a1-kube-api-access-h4tm8\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:57:10.931416 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:10.931388    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh_f1a24661-6e05-4139-8b73-f54fa2fac3a1/storage-initializer/1.log"
Jun 15 06:57:10.931901 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:10.931501    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh"
Jun 15 06:57:10.931901 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:10.931515    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh" event={"ID":"f1a24661-6e05-4139-8b73-f54fa2fac3a1","Type":"ContainerDied","Data":"716a92ef1f21ee5b144ac435c612d655986af08e68fa3a8c15386a351aa53e09"}
Jun 15 06:57:10.931901 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:10.931570    2572 scope.go:117] "RemoveContainer" containerID="b24ab4fd9af791b08c2f088339af14a5c745a65f4e6458190233235e5f2f01b3"
Jun 15 06:57:10.990092 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:10.990063    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh"]
Jun 15 06:57:10.995066 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:10.995040    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-c9b96cfdf-dhfgh"]
Jun 15 06:57:11.323927 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:11.323836    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1a24661-6e05-4139-8b73-f54fa2fac3a1" path="/var/lib/kubelet/pods/f1a24661-6e05-4139-8b73-f54fa2fac3a1/volumes"
Jun 15 06:57:15.401385 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.401337    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-s4g7x/must-gather-5kh2n"]
Jun 15 06:57:15.401922 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.401901    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f4aa51-b485-41b0-955d-fb91faef78da" containerName="kserve-container"
Jun 15 06:57:15.402027 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.401926    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f4aa51-b485-41b0-955d-fb91faef78da" containerName="kserve-container"
Jun 15 06:57:15.402027 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.401942    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f4aa51-b485-41b0-955d-fb91faef78da" containerName="kube-rbac-proxy"
Jun 15 06:57:15.402027 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.401949    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f4aa51-b485-41b0-955d-fb91faef78da" containerName="kube-rbac-proxy"
Jun 15 06:57:15.402027 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.401991    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1a24661-6e05-4139-8b73-f54fa2fac3a1" containerName="storage-initializer"
Jun 15 06:57:15.402027 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.402000    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1a24661-6e05-4139-8b73-f54fa2fac3a1" containerName="storage-initializer"
Jun 15 06:57:15.402027 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.402010    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1a24661-6e05-4139-8b73-f54fa2fac3a1" containerName="storage-initializer"
Jun 15 06:57:15.402027 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.402019    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1a24661-6e05-4139-8b73-f54fa2fac3a1" containerName="storage-initializer"
Jun 15 06:57:15.402421 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.402031    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2f4aa51-b485-41b0-955d-fb91faef78da" containerName="storage-initializer"
Jun 15 06:57:15.402421 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.402041    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2f4aa51-b485-41b0-955d-fb91faef78da" containerName="storage-initializer"
Jun 15 06:57:15.402421 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.402132    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1a24661-6e05-4139-8b73-f54fa2fac3a1" containerName="storage-initializer"
Jun 15 06:57:15.402421 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.402145    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2f4aa51-b485-41b0-955d-fb91faef78da" containerName="kube-rbac-proxy"
Jun 15 06:57:15.402421 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.402158    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2f4aa51-b485-41b0-955d-fb91faef78da" containerName="kserve-container"
Jun 15 06:57:15.402421 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.402320    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1a24661-6e05-4139-8b73-f54fa2fac3a1" containerName="storage-initializer"
Jun 15 06:57:15.405635 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.405615    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s4g7x/must-gather-5kh2n"
Jun 15 06:57:15.408558 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.408542    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s4g7x\"/\"kube-root-ca.crt\""
Jun 15 06:57:15.408999 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.408986    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-s4g7x\"/\"openshift-service-ca.crt\""
Jun 15 06:57:15.409416 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.409400    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-s4g7x\"/\"default-dockercfg-zq5pj\""
Jun 15 06:57:15.424577 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.424554    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s4g7x/must-gather-5kh2n"]
Jun 15 06:57:15.517469 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.517424    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnbf8\" (UniqueName: \"kubernetes.io/projected/21405b6b-5a2e-4e88-a61b-27d9327f3aee-kube-api-access-xnbf8\") pod \"must-gather-5kh2n\" (UID: \"21405b6b-5a2e-4e88-a61b-27d9327f3aee\") " pod="openshift-must-gather-s4g7x/must-gather-5kh2n"
Jun 15 06:57:15.517469 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.517472    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/21405b6b-5a2e-4e88-a61b-27d9327f3aee-must-gather-output\") pod \"must-gather-5kh2n\" (UID: \"21405b6b-5a2e-4e88-a61b-27d9327f3aee\") " pod="openshift-must-gather-s4g7x/must-gather-5kh2n"
Jun 15 06:57:15.618164 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.618123    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnbf8\" (UniqueName: \"kubernetes.io/projected/21405b6b-5a2e-4e88-a61b-27d9327f3aee-kube-api-access-xnbf8\") pod \"must-gather-5kh2n\" (UID: \"21405b6b-5a2e-4e88-a61b-27d9327f3aee\") " pod="openshift-must-gather-s4g7x/must-gather-5kh2n"
Jun 15 06:57:15.618322 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.618169    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/21405b6b-5a2e-4e88-a61b-27d9327f3aee-must-gather-output\") pod \"must-gather-5kh2n\" (UID: \"21405b6b-5a2e-4e88-a61b-27d9327f3aee\") " pod="openshift-must-gather-s4g7x/must-gather-5kh2n"
Jun 15 06:57:15.618580 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.618557    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/21405b6b-5a2e-4e88-a61b-27d9327f3aee-must-gather-output\") pod \"must-gather-5kh2n\" (UID: \"21405b6b-5a2e-4e88-a61b-27d9327f3aee\") " pod="openshift-must-gather-s4g7x/must-gather-5kh2n"
Jun 15 06:57:15.626727 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.626708    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnbf8\" (UniqueName: \"kubernetes.io/projected/21405b6b-5a2e-4e88-a61b-27d9327f3aee-kube-api-access-xnbf8\") pod \"must-gather-5kh2n\" (UID: \"21405b6b-5a2e-4e88-a61b-27d9327f3aee\") " pod="openshift-must-gather-s4g7x/must-gather-5kh2n"
Jun 15 06:57:15.733642 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.733561    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s4g7x/must-gather-5kh2n"
Jun 15 06:57:15.863626 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.863599    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-s4g7x/must-gather-5kh2n"]
Jun 15 06:57:15.866220 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:57:15.866194    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21405b6b_5a2e_4e88_a61b_27d9327f3aee.slice/crio-504aea6568f6a4b61daf05a0ae22f57b66f49868a98bcbf9cf01330b34175947 WatchSource:0}: Error finding container 504aea6568f6a4b61daf05a0ae22f57b66f49868a98bcbf9cf01330b34175947: Status 404 returned error can't find the container with id 504aea6568f6a4b61daf05a0ae22f57b66f49868a98bcbf9cf01330b34175947
Jun 15 06:57:15.867985 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.867965    2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jun 15 06:57:15.959609 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:15.959576    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s4g7x/must-gather-5kh2n" event={"ID":"21405b6b-5a2e-4e88-a61b-27d9327f3aee","Type":"ContainerStarted","Data":"504aea6568f6a4b61daf05a0ae22f57b66f49868a98bcbf9cf01330b34175947"}
Jun 15 06:57:23.009317 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:23.009285    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s4g7x/must-gather-5kh2n" event={"ID":"21405b6b-5a2e-4e88-a61b-27d9327f3aee","Type":"ContainerStarted","Data":"f340edeca49b9f9c575b0899a6c5df91354d414be424c0d8ca6c41d3324521c7"}
Jun 15 06:57:23.009793 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:23.009330    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s4g7x/must-gather-5kh2n" event={"ID":"21405b6b-5a2e-4e88-a61b-27d9327f3aee","Type":"ContainerStarted","Data":"1b575008d1ba066dfa3f3704b0e4af960790b4b75e5dc5d20d1e6956a8e3d058"}
Jun 15 06:57:23.026880 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:23.026832    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-s4g7x/must-gather-5kh2n" podStartSLOduration=1.987457435 podStartE2EDuration="8.026815426s" totalImagesPullingTime="6.039357991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:57:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:57:23.025449258 +0000 UTC m=+4158.173629257" watchObservedRunningTime="2026-06-15 06:57:23.026815426 +0000 UTC m=+4158.174995423"
Jun 15 06:57:44.114686 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:44.114594    2572 generic.go:356] "Generic (PLEG): container finished" podID="21405b6b-5a2e-4e88-a61b-27d9327f3aee" containerID="1b575008d1ba066dfa3f3704b0e4af960790b4b75e5dc5d20d1e6956a8e3d058" exitCode=0
Jun 15 06:57:44.115190 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:44.114673    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-s4g7x/must-gather-5kh2n" event={"ID":"21405b6b-5a2e-4e88-a61b-27d9327f3aee","Type":"ContainerDied","Data":"1b575008d1ba066dfa3f3704b0e4af960790b4b75e5dc5d20d1e6956a8e3d058"}
Jun 15 06:57:44.115190 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:44.115100    2572 scope.go:117] "RemoveContainer" containerID="1b575008d1ba066dfa3f3704b0e4af960790b4b75e5dc5d20d1e6956a8e3d058"
Jun 15 06:57:44.892747 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:44.892717    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-s4g7x_must-gather-5kh2n_21405b6b-5a2e-4e88-a61b-27d9327f3aee/gather/0.log"
Jun 15 06:57:45.558977 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:45.558942    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w2k28/must-gather-7fbrw"]
Jun 15 06:57:45.561468 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:45.561448    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w2k28/must-gather-7fbrw"
Jun 15 06:57:45.563583 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:45.563560    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w2k28\"/\"kube-root-ca.crt\""
Jun 15 06:57:45.563709 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:45.563562    2572 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-w2k28\"/\"default-dockercfg-5kg2k\""
Jun 15 06:57:45.563709 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:45.563591    2572 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-w2k28\"/\"openshift-service-ca.crt\""
Jun 15 06:57:45.569709 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:45.569687    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w2k28/must-gather-7fbrw"]
Jun 15 06:57:45.699462 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:45.699420    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw86h\" (UniqueName: \"kubernetes.io/projected/8a3c6342-3400-4857-92ac-d5f3a98a6cd4-kube-api-access-mw86h\") pod \"must-gather-7fbrw\" (UID: \"8a3c6342-3400-4857-92ac-d5f3a98a6cd4\") " pod="openshift-must-gather-w2k28/must-gather-7fbrw"
Jun 15 06:57:45.699633 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:45.699569    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8a3c6342-3400-4857-92ac-d5f3a98a6cd4-must-gather-output\") pod \"must-gather-7fbrw\" (UID: \"8a3c6342-3400-4857-92ac-d5f3a98a6cd4\") " pod="openshift-must-gather-w2k28/must-gather-7fbrw"
Jun 15 06:57:45.800197 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:45.800158    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8a3c6342-3400-4857-92ac-d5f3a98a6cd4-must-gather-output\") pod \"must-gather-7fbrw\" (UID: \"8a3c6342-3400-4857-92ac-d5f3a98a6cd4\") " pod="openshift-must-gather-w2k28/must-gather-7fbrw"
Jun 15 06:57:45.800398 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:45.800220    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw86h\" (UniqueName: \"kubernetes.io/projected/8a3c6342-3400-4857-92ac-d5f3a98a6cd4-kube-api-access-mw86h\") pod \"must-gather-7fbrw\" (UID: \"8a3c6342-3400-4857-92ac-d5f3a98a6cd4\") " pod="openshift-must-gather-w2k28/must-gather-7fbrw"
Jun 15 06:57:45.800596 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:45.800573    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8a3c6342-3400-4857-92ac-d5f3a98a6cd4-must-gather-output\") pod \"must-gather-7fbrw\" (UID: \"8a3c6342-3400-4857-92ac-d5f3a98a6cd4\") " pod="openshift-must-gather-w2k28/must-gather-7fbrw"
Jun 15 06:57:45.809829 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:45.809771    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw86h\" (UniqueName: \"kubernetes.io/projected/8a3c6342-3400-4857-92ac-d5f3a98a6cd4-kube-api-access-mw86h\") pod \"must-gather-7fbrw\" (UID: \"8a3c6342-3400-4857-92ac-d5f3a98a6cd4\") " pod="openshift-must-gather-w2k28/must-gather-7fbrw"
Jun 15 06:57:45.871420 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:45.871387    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w2k28/must-gather-7fbrw"
Jun 15 06:57:45.998415 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:45.998391    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w2k28/must-gather-7fbrw"]
Jun 15 06:57:46.000598 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:57:46.000562    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a3c6342_3400_4857_92ac_d5f3a98a6cd4.slice/crio-a1b8fb818820fc88d50758e5f350094096513e3236124e0b8ac91671d2faaa43 WatchSource:0}: Error finding container a1b8fb818820fc88d50758e5f350094096513e3236124e0b8ac91671d2faaa43: Status 404 returned error can't find the container with id a1b8fb818820fc88d50758e5f350094096513e3236124e0b8ac91671d2faaa43
Jun 15 06:57:46.124673 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:46.124589    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w2k28/must-gather-7fbrw" event={"ID":"8a3c6342-3400-4857-92ac-d5f3a98a6cd4","Type":"ContainerStarted","Data":"a1b8fb818820fc88d50758e5f350094096513e3236124e0b8ac91671d2faaa43"}
Jun 15 06:57:47.132331 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:47.132231    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w2k28/must-gather-7fbrw" event={"ID":"8a3c6342-3400-4857-92ac-d5f3a98a6cd4","Type":"ContainerStarted","Data":"717fc1f4b8902ff71c13050ca92cbdc202845931cec25886a3c074092e61fc7d"}
Jun 15 06:57:47.132331 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:47.132283    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w2k28/must-gather-7fbrw" event={"ID":"8a3c6342-3400-4857-92ac-d5f3a98a6cd4","Type":"ContainerStarted","Data":"eb4f44656be3e6389d5b8674a9e8d35841d13c27de68420ae14a1919631d6c08"}
Jun 15 06:57:47.152314 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:47.152260    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-w2k28/must-gather-7fbrw" podStartSLOduration=1.418321039 podStartE2EDuration="2.152242985s" totalImagesPullingTime="733.921946ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-06-15 06:57:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:57:47.150216345 +0000 UTC m=+4182.298396336" watchObservedRunningTime="2026-06-15 06:57:47.152242985 +0000 UTC m=+4182.300422982"
Jun 15 06:57:48.432818 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:48.432782    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-z9ws7_8c764213-65e7-4296-a8b6-023c581b36c1/global-pull-secret-syncer/0.log"
Jun 15 06:57:48.560438 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:48.560414    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-kxfln_60d38c85-2ead-4253-915b-fa8462546084/konnectivity-agent/0.log"
Jun 15 06:57:48.644219 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:48.644186    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-16.ec2.internal_f55c91d0f7b296b874c42e3338400df2/haproxy/0.log"
Jun 15 06:57:50.410575 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:50.410514    2572 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-s4g7x/must-gather-5kh2n"]
Jun 15 06:57:50.411563 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:50.411526    2572 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-s4g7x/must-gather-5kh2n" podUID="21405b6b-5a2e-4e88-a61b-27d9327f3aee" containerName="copy" containerID="cri-o://f340edeca49b9f9c575b0899a6c5df91354d414be424c0d8ca6c41d3324521c7" gracePeriod=2
Jun 15 06:57:50.416433 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:57:50.416317    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-5kh2n\" is forbidden: User \"system:node:ip-10-0-138-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-s4g7x\": no relationship found between node 'ip-10-0-138-16.ec2.internal' and this object" podUID="21405b6b-5a2e-4e88-a61b-27d9327f3aee" pod="openshift-must-gather-s4g7x/must-gather-5kh2n"
Jun 15 06:57:50.417723 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:50.417700    2572 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-s4g7x/must-gather-5kh2n"]
Jun 15 06:57:50.795379 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:50.795018    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-s4g7x_must-gather-5kh2n_21405b6b-5a2e-4e88-a61b-27d9327f3aee/copy/0.log"
Jun 15 06:57:50.799379 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:50.798322    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s4g7x/must-gather-5kh2n"
Jun 15 06:57:50.802149 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:57:50.802108    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-5kh2n\" is forbidden: User \"system:node:ip-10-0-138-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-s4g7x\": no relationship found between node 'ip-10-0-138-16.ec2.internal' and this object" podUID="21405b6b-5a2e-4e88-a61b-27d9327f3aee" pod="openshift-must-gather-s4g7x/must-gather-5kh2n"
Jun 15 06:57:50.954786 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:50.954752    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xnbf8\" (UniqueName: \"kubernetes.io/projected/21405b6b-5a2e-4e88-a61b-27d9327f3aee-kube-api-access-xnbf8\") pod \"21405b6b-5a2e-4e88-a61b-27d9327f3aee\" (UID: \"21405b6b-5a2e-4e88-a61b-27d9327f3aee\") "
Jun 15 06:57:50.955115 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:50.955093    2572 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/21405b6b-5a2e-4e88-a61b-27d9327f3aee-must-gather-output\") pod \"21405b6b-5a2e-4e88-a61b-27d9327f3aee\" (UID: \"21405b6b-5a2e-4e88-a61b-27d9327f3aee\") "
Jun 15 06:57:50.956986 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:50.956955    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21405b6b-5a2e-4e88-a61b-27d9327f3aee-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "21405b6b-5a2e-4e88-a61b-27d9327f3aee" (UID: "21405b6b-5a2e-4e88-a61b-27d9327f3aee"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jun 15 06:57:50.968848 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:50.968821    2572 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21405b6b-5a2e-4e88-a61b-27d9327f3aee-kube-api-access-xnbf8" (OuterVolumeSpecName: "kube-api-access-xnbf8") pod "21405b6b-5a2e-4e88-a61b-27d9327f3aee" (UID: "21405b6b-5a2e-4e88-a61b-27d9327f3aee"). InnerVolumeSpecName "kube-api-access-xnbf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jun 15 06:57:51.056850 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:51.056765    2572 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnbf8\" (UniqueName: \"kubernetes.io/projected/21405b6b-5a2e-4e88-a61b-27d9327f3aee-kube-api-access-xnbf8\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:57:51.057118 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:51.057098    2572 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/21405b6b-5a2e-4e88-a61b-27d9327f3aee-must-gather-output\") on node \"ip-10-0-138-16.ec2.internal\" DevicePath \"\""
Jun 15 06:57:51.157968 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:51.157937    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-s4g7x_must-gather-5kh2n_21405b6b-5a2e-4e88-a61b-27d9327f3aee/copy/0.log"
Jun 15 06:57:51.159808 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:51.159781    2572 generic.go:356] "Generic (PLEG): container finished" podID="21405b6b-5a2e-4e88-a61b-27d9327f3aee" containerID="f340edeca49b9f9c575b0899a6c5df91354d414be424c0d8ca6c41d3324521c7" exitCode=143
Jun 15 06:57:51.161133 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:51.160005    2572 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-s4g7x/must-gather-5kh2n"
Jun 15 06:57:51.165840 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:51.160037    2572 scope.go:117] "RemoveContainer" containerID="f340edeca49b9f9c575b0899a6c5df91354d414be424c0d8ca6c41d3324521c7"
Jun 15 06:57:51.168577 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:57:51.168552    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-5kh2n\" is forbidden: User \"system:node:ip-10-0-138-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-s4g7x\": no relationship found between node 'ip-10-0-138-16.ec2.internal' and this object" podUID="21405b6b-5a2e-4e88-a61b-27d9327f3aee" pod="openshift-must-gather-s4g7x/must-gather-5kh2n"
Jun 15 06:57:51.179156 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:51.179142    2572 scope.go:117] "RemoveContainer" containerID="1b575008d1ba066dfa3f3704b0e4af960790b4b75e5dc5d20d1e6956a8e3d058"
Jun 15 06:57:51.184817 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:57:51.184772    2572 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-5kh2n\" is forbidden: User \"system:node:ip-10-0-138-16.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-s4g7x\": no relationship found between node 'ip-10-0-138-16.ec2.internal' and this object" podUID="21405b6b-5a2e-4e88-a61b-27d9327f3aee" pod="openshift-must-gather-s4g7x/must-gather-5kh2n"
Jun 15 06:57:51.202547 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:51.202520    2572 scope.go:117] "RemoveContainer" containerID="f340edeca49b9f9c575b0899a6c5df91354d414be424c0d8ca6c41d3324521c7"
Jun 15 06:57:51.203150 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:57:51.203126    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f340edeca49b9f9c575b0899a6c5df91354d414be424c0d8ca6c41d3324521c7\": container with ID starting with f340edeca49b9f9c575b0899a6c5df91354d414be424c0d8ca6c41d3324521c7 not found: ID does not exist" containerID="f340edeca49b9f9c575b0899a6c5df91354d414be424c0d8ca6c41d3324521c7"
Jun 15 06:57:51.203302 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:51.203277    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f340edeca49b9f9c575b0899a6c5df91354d414be424c0d8ca6c41d3324521c7"} err="failed to get container status \"f340edeca49b9f9c575b0899a6c5df91354d414be424c0d8ca6c41d3324521c7\": rpc error: code = NotFound desc = could not find container \"f340edeca49b9f9c575b0899a6c5df91354d414be424c0d8ca6c41d3324521c7\": container with ID starting with f340edeca49b9f9c575b0899a6c5df91354d414be424c0d8ca6c41d3324521c7 not found: ID does not exist"
Jun 15 06:57:51.203417 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:51.203405    2572 scope.go:117] "RemoveContainer" containerID="1b575008d1ba066dfa3f3704b0e4af960790b4b75e5dc5d20d1e6956a8e3d058"
Jun 15 06:57:51.203809 ip-10-0-138-16 kubenswrapper[2572]: E0615 06:57:51.203790    2572 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b575008d1ba066dfa3f3704b0e4af960790b4b75e5dc5d20d1e6956a8e3d058\": container with ID starting with 1b575008d1ba066dfa3f3704b0e4af960790b4b75e5dc5d20d1e6956a8e3d058 not found: ID does not exist" containerID="1b575008d1ba066dfa3f3704b0e4af960790b4b75e5dc5d20d1e6956a8e3d058"
Jun 15 06:57:51.203940 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:51.203919    2572 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b575008d1ba066dfa3f3704b0e4af960790b4b75e5dc5d20d1e6956a8e3d058"} err="failed to get container status \"1b575008d1ba066dfa3f3704b0e4af960790b4b75e5dc5d20d1e6956a8e3d058\": rpc error: code = NotFound desc = could not find container \"1b575008d1ba066dfa3f3704b0e4af960790b4b75e5dc5d20d1e6956a8e3d058\": container with ID starting with 1b575008d1ba066dfa3f3704b0e4af960790b4b75e5dc5d20d1e6956a8e3d058 not found: ID does not exist"
Jun 15 06:57:51.326907 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:51.326830    2572 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21405b6b-5a2e-4e88-a61b-27d9327f3aee" path="/var/lib/kubelet/pods/21405b6b-5a2e-4e88-a61b-27d9327f3aee/volumes"
Jun 15 06:57:52.947537 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:52.947504    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_60712149-ddd7-40bc-8c65-bbccb02d229f/alertmanager/0.log"
Jun 15 06:57:52.986989 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:52.986907    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_60712149-ddd7-40bc-8c65-bbccb02d229f/config-reloader/0.log"
Jun 15 06:57:53.014634 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:53.014571    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_60712149-ddd7-40bc-8c65-bbccb02d229f/kube-rbac-proxy-web/0.log"
Jun 15 06:57:53.043077 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:53.043004    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_60712149-ddd7-40bc-8c65-bbccb02d229f/kube-rbac-proxy/0.log"
Jun 15 06:57:53.071614 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:53.071586    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_60712149-ddd7-40bc-8c65-bbccb02d229f/kube-rbac-proxy-metric/0.log"
Jun 15 06:57:53.100399 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:53.100354    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_60712149-ddd7-40bc-8c65-bbccb02d229f/prom-label-proxy/0.log"
Jun 15 06:57:53.126829 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:53.126793    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_60712149-ddd7-40bc-8c65-bbccb02d229f/init-config-reloader/0.log"
Jun 15 06:57:53.208113 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:53.208042    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-678bc549d7-7746w_c7af7387-0238-4b94-9a45-8bcaa37446fd/kube-state-metrics/0.log"
Jun 15 06:57:53.235673 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:53.235639    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-678bc549d7-7746w_c7af7387-0238-4b94-9a45-8bcaa37446fd/kube-rbac-proxy-main/0.log"
Jun 15 06:57:53.262774 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:53.262736    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-678bc549d7-7746w_c7af7387-0238-4b94-9a45-8bcaa37446fd/kube-rbac-proxy-self/0.log"
Jun 15 06:57:53.305077 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:53.305038    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-544cd4646d-gzxjx_92ef9a64-4060-4a86-a206-d299de7e4b80/metrics-server/0.log"
Jun 15 06:57:53.335596 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:53.335567    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-589576dfc7-pqjxp_1ba3283e-aed0-44dd-bbd3-daca09b7cfc5/monitoring-plugin/0.log"
Jun 15 06:57:53.370258 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:53.370216    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9hzhf_24202341-f48a-4a16-9619-eab661b11dc3/node-exporter/0.log"
Jun 15 06:57:53.399275 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:53.399249    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9hzhf_24202341-f48a-4a16-9619-eab661b11dc3/kube-rbac-proxy/0.log"
Jun 15 06:57:53.426190 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:53.426162    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9hzhf_24202341-f48a-4a16-9619-eab661b11dc3/init-textfile/0.log"
Jun 15 06:57:53.637395 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:53.637344    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6b8d965dd5-x6l2f_17ffae6f-c8ca-4b19-bd6a-20a89393597b/kube-rbac-proxy-main/0.log"
Jun 15 06:57:53.676345 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:53.676311    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6b8d965dd5-x6l2f_17ffae6f-c8ca-4b19-bd6a-20a89393597b/kube-rbac-proxy-self/0.log"
Jun 15 06:57:53.701931 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:53.701736    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-6b8d965dd5-x6l2f_17ffae6f-c8ca-4b19-bd6a-20a89393597b/openshift-state-metrics/0.log"
Jun 15 06:57:53.748210 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:53.748178    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3fb0e7d5-7654-4de4-8a00-60f36fb6c64f/prometheus/0.log"
Jun 15 06:57:53.771703 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:53.771671    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3fb0e7d5-7654-4de4-8a00-60f36fb6c64f/config-reloader/0.log"
Jun 15 06:57:53.797821 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:53.797777    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3fb0e7d5-7654-4de4-8a00-60f36fb6c64f/thanos-sidecar/0.log"
Jun 15 06:57:53.823512 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:53.823482    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3fb0e7d5-7654-4de4-8a00-60f36fb6c64f/kube-rbac-proxy-web/0.log"
Jun 15 06:57:53.847703 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:53.847672    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3fb0e7d5-7654-4de4-8a00-60f36fb6c64f/kube-rbac-proxy/0.log"
Jun 15 06:57:53.876544 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:53.876465    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3fb0e7d5-7654-4de4-8a00-60f36fb6c64f/kube-rbac-proxy-thanos/0.log"
Jun 15 06:57:53.903892 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:53.903868    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3fb0e7d5-7654-4de4-8a00-60f36fb6c64f/init-config-reloader/0.log"
Jun 15 06:57:53.936990 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:53.936957    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-864fcfc6f9-54pzw_96f776da-6137-4f8b-8324-e6cab29e36d6/prometheus-operator/0.log"
Jun 15 06:57:53.959040 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:53.959012    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-864fcfc6f9-54pzw_96f776da-6137-4f8b-8324-e6cab29e36d6/kube-rbac-proxy/0.log"
Jun 15 06:57:54.029428 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:54.029399    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5cf6999874-hqcvz_4d27f524-03f3-45ca-8026-9880e1cfc205/telemeter-client/0.log"
Jun 15 06:57:54.052973 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:54.052945    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5cf6999874-hqcvz_4d27f524-03f3-45ca-8026-9880e1cfc205/reload/0.log"
Jun 15 06:57:54.081319 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:54.081293    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5cf6999874-hqcvz_4d27f524-03f3-45ca-8026-9880e1cfc205/kube-rbac-proxy/0.log"
Jun 15 06:57:54.113228 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:54.113201    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-569bcc98cc-w6242_d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef/thanos-query/0.log"
Jun 15 06:57:54.140915 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:54.140880    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-569bcc98cc-w6242_d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef/kube-rbac-proxy-web/0.log"
Jun 15 06:57:54.168550 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:54.168520    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-569bcc98cc-w6242_d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef/kube-rbac-proxy/0.log"
Jun 15 06:57:54.194889 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:54.194852    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-569bcc98cc-w6242_d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef/prom-label-proxy/0.log"
Jun 15 06:57:54.219468 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:54.219439    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-569bcc98cc-w6242_d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef/kube-rbac-proxy-rules/0.log"
Jun 15 06:57:54.246831 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:54.246764    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-569bcc98cc-w6242_d8019fb9-62b7-4c1f-bf6d-c42da66dc6ef/kube-rbac-proxy-metrics/0.log"
Jun 15 06:57:56.486731 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:56.486691    2572 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-w2k28/perf-node-gather-daemonset-bf7bn"]
Jun 15 06:57:56.487295 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:56.487240    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21405b6b-5a2e-4e88-a61b-27d9327f3aee" containerName="gather"
Jun 15 06:57:56.487295 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:56.487260    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="21405b6b-5a2e-4e88-a61b-27d9327f3aee" containerName="gather"
Jun 15 06:57:56.487295 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:56.487276    2572 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21405b6b-5a2e-4e88-a61b-27d9327f3aee" containerName="copy"
Jun 15 06:57:56.487295 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:56.487285    2572 state_mem.go:107] "Deleted CPUSet assignment" podUID="21405b6b-5a2e-4e88-a61b-27d9327f3aee" containerName="copy"
Jun 15 06:57:56.487573 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:56.487413    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="21405b6b-5a2e-4e88-a61b-27d9327f3aee" containerName="gather"
Jun 15 06:57:56.487573 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:56.487430    2572 memory_manager.go:368] "RemoveStaleState removing state" podUID="21405b6b-5a2e-4e88-a61b-27d9327f3aee" containerName="copy"
Jun 15 06:57:56.492922 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:56.492897    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w2k28/perf-node-gather-daemonset-bf7bn"
Jun 15 06:57:56.504850 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:56.504819    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w2k28/perf-node-gather-daemonset-bf7bn"]
Jun 15 06:57:56.613838 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:56.613798    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k2tm\" (UniqueName: \"kubernetes.io/projected/156cea6c-9cb3-481a-b5d8-f9c96ceef943-kube-api-access-9k2tm\") pod \"perf-node-gather-daemonset-bf7bn\" (UID: \"156cea6c-9cb3-481a-b5d8-f9c96ceef943\") " pod="openshift-must-gather-w2k28/perf-node-gather-daemonset-bf7bn"
Jun 15 06:57:56.613999 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:56.613857    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/156cea6c-9cb3-481a-b5d8-f9c96ceef943-lib-modules\") pod \"perf-node-gather-daemonset-bf7bn\" (UID: \"156cea6c-9cb3-481a-b5d8-f9c96ceef943\") " pod="openshift-must-gather-w2k28/perf-node-gather-daemonset-bf7bn"
Jun 15 06:57:56.613999 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:56.613894    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/156cea6c-9cb3-481a-b5d8-f9c96ceef943-sys\") pod \"perf-node-gather-daemonset-bf7bn\" (UID: \"156cea6c-9cb3-481a-b5d8-f9c96ceef943\") " pod="openshift-must-gather-w2k28/perf-node-gather-daemonset-bf7bn"
Jun 15 06:57:56.613999 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:56.613922    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/156cea6c-9cb3-481a-b5d8-f9c96ceef943-podres\") pod \"perf-node-gather-daemonset-bf7bn\" (UID: \"156cea6c-9cb3-481a-b5d8-f9c96ceef943\") " pod="openshift-must-gather-w2k28/perf-node-gather-daemonset-bf7bn"
Jun 15 06:57:56.613999 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:56.613965    2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/156cea6c-9cb3-481a-b5d8-f9c96ceef943-proc\") pod \"perf-node-gather-daemonset-bf7bn\" (UID: \"156cea6c-9cb3-481a-b5d8-f9c96ceef943\") " pod="openshift-must-gather-w2k28/perf-node-gather-daemonset-bf7bn"
Jun 15 06:57:56.646702 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:56.646673    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-866d768b47-vq4j7_27b5148e-64d5-4397-bb36-d031b4d146e8/console/0.log"
Jun 15 06:57:56.714972 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:56.714930    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k2tm\" (UniqueName: \"kubernetes.io/projected/156cea6c-9cb3-481a-b5d8-f9c96ceef943-kube-api-access-9k2tm\") pod \"perf-node-gather-daemonset-bf7bn\" (UID: \"156cea6c-9cb3-481a-b5d8-f9c96ceef943\") " pod="openshift-must-gather-w2k28/perf-node-gather-daemonset-bf7bn"
Jun 15 06:57:56.715167 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:56.714988    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/156cea6c-9cb3-481a-b5d8-f9c96ceef943-lib-modules\") pod \"perf-node-gather-daemonset-bf7bn\" (UID: \"156cea6c-9cb3-481a-b5d8-f9c96ceef943\") " pod="openshift-must-gather-w2k28/perf-node-gather-daemonset-bf7bn"
Jun 15 06:57:56.715167 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:56.715026    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/156cea6c-9cb3-481a-b5d8-f9c96ceef943-sys\") pod \"perf-node-gather-daemonset-bf7bn\" (UID: \"156cea6c-9cb3-481a-b5d8-f9c96ceef943\") " pod="openshift-must-gather-w2k28/perf-node-gather-daemonset-bf7bn"
Jun 15 06:57:56.715167 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:56.715057    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/156cea6c-9cb3-481a-b5d8-f9c96ceef943-podres\") pod \"perf-node-gather-daemonset-bf7bn\" (UID: \"156cea6c-9cb3-481a-b5d8-f9c96ceef943\") " pod="openshift-must-gather-w2k28/perf-node-gather-daemonset-bf7bn"
Jun 15 06:57:56.715167 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:56.715100    2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/156cea6c-9cb3-481a-b5d8-f9c96ceef943-proc\") pod \"perf-node-gather-daemonset-bf7bn\" (UID: \"156cea6c-9cb3-481a-b5d8-f9c96ceef943\") " pod="openshift-must-gather-w2k28/perf-node-gather-daemonset-bf7bn"
Jun 15 06:57:56.715434 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:56.715190    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/156cea6c-9cb3-481a-b5d8-f9c96ceef943-lib-modules\") pod \"perf-node-gather-daemonset-bf7bn\" (UID: \"156cea6c-9cb3-481a-b5d8-f9c96ceef943\") " pod="openshift-must-gather-w2k28/perf-node-gather-daemonset-bf7bn"
Jun 15 06:57:56.715434 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:56.715201    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/156cea6c-9cb3-481a-b5d8-f9c96ceef943-proc\") pod \"perf-node-gather-daemonset-bf7bn\" (UID: \"156cea6c-9cb3-481a-b5d8-f9c96ceef943\") " pod="openshift-must-gather-w2k28/perf-node-gather-daemonset-bf7bn"
Jun 15 06:57:56.715434 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:56.715241    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/156cea6c-9cb3-481a-b5d8-f9c96ceef943-sys\") pod \"perf-node-gather-daemonset-bf7bn\" (UID: \"156cea6c-9cb3-481a-b5d8-f9c96ceef943\") " pod="openshift-must-gather-w2k28/perf-node-gather-daemonset-bf7bn"
Jun 15 06:57:56.715434 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:56.715305    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/156cea6c-9cb3-481a-b5d8-f9c96ceef943-podres\") pod \"perf-node-gather-daemonset-bf7bn\" (UID: \"156cea6c-9cb3-481a-b5d8-f9c96ceef943\") " pod="openshift-must-gather-w2k28/perf-node-gather-daemonset-bf7bn"
Jun 15 06:57:56.726147 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:56.726104    2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k2tm\" (UniqueName: \"kubernetes.io/projected/156cea6c-9cb3-481a-b5d8-f9c96ceef943-kube-api-access-9k2tm\") pod \"perf-node-gather-daemonset-bf7bn\" (UID: \"156cea6c-9cb3-481a-b5d8-f9c96ceef943\") " pod="openshift-must-gather-w2k28/perf-node-gather-daemonset-bf7bn"
Jun 15 06:57:56.811188 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:56.811096    2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-w2k28/perf-node-gather-daemonset-bf7bn"
Jun 15 06:57:56.963466 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:56.963440    2572 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-w2k28/perf-node-gather-daemonset-bf7bn"]
Jun 15 06:57:56.966023 ip-10-0-138-16 kubenswrapper[2572]: W0615 06:57:56.965992    2572 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod156cea6c_9cb3_481a_b5d8_f9c96ceef943.slice/crio-888219cbbbcaaa300487d6eb847a3ab6c0848b4c963c3d10471978d6535bb417 WatchSource:0}: Error finding container 888219cbbbcaaa300487d6eb847a3ab6c0848b4c963c3d10471978d6535bb417: Status 404 returned error can't find the container with id 888219cbbbcaaa300487d6eb847a3ab6c0848b4c963c3d10471978d6535bb417
Jun 15 06:57:57.214027 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:57.213985    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w2k28/perf-node-gather-daemonset-bf7bn" event={"ID":"156cea6c-9cb3-481a-b5d8-f9c96ceef943","Type":"ContainerStarted","Data":"f6430d32b4a1b8bfdddc23bd14d1b964527fd628fd501ce47be2ea3bee38e49a"}
Jun 15 06:57:57.214210 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:57.214057    2572 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-w2k28/perf-node-gather-daemonset-bf7bn" event={"ID":"156cea6c-9cb3-481a-b5d8-f9c96ceef943","Type":"ContainerStarted","Data":"888219cbbbcaaa300487d6eb847a3ab6c0848b4c963c3d10471978d6535bb417"}
Jun 15 06:57:57.214210 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:57.214087    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-w2k28/perf-node-gather-daemonset-bf7bn"
Jun 15 06:57:57.232026 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:57.231958    2572 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-w2k28/perf-node-gather-daemonset-bf7bn" podStartSLOduration=1.23193851 podStartE2EDuration="1.23193851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-06-15 06:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-06-15 06:57:57.230859595 +0000 UTC m=+4192.379039593" watchObservedRunningTime="2026-06-15 06:57:57.23193851 +0000 UTC m=+4192.380118508"
Jun 15 06:57:58.187012 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:58.186981    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q6c87_47e37b20-ad06-4ff6-9beb-6503464f6ef8/dns/0.log"
Jun 15 06:57:58.211077 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:58.211043    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q6c87_47e37b20-ad06-4ff6-9beb-6503464f6ef8/kube-rbac-proxy/0.log"
Jun 15 06:57:58.337141 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:58.337114    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4dv8m_54e26677-5db1-4e31-94c1-a5d3c5db5047/dns-node-resolver/0.log"
Jun 15 06:57:58.958830 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:57:58.958801    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g9tgf_c80610bc-c188-476f-8a0c-ce642967f28e/node-ca/0.log"
Jun 15 06:58:00.140276 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:00.140245    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-k5l48_6f7e3592-7192-4acb-a079-7cc24646f8a5/serve-healthcheck-canary/0.log"
Jun 15 06:58:00.721387 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:00.721340    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xpdg2_bdbef6be-323c-4a82-9f0a-8bb8fdafc620/kube-rbac-proxy/0.log"
Jun 15 06:58:00.745052 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:00.745025    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xpdg2_bdbef6be-323c-4a82-9f0a-8bb8fdafc620/exporter/0.log"
Jun 15 06:58:00.771300 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:00.771274    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-xpdg2_bdbef6be-323c-4a82-9f0a-8bb8fdafc620/extractor/0.log"
Jun 15 06:58:02.934819 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:02.934792    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-zt9mh_ba044465-40ed-4fe8-ab1a-fee1bc6065c9/server/0.log"
Jun 15 06:58:03.172368 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:03.172323    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-vqqhs_93fa76e3-706a-4b9a-bc8d-2ffb5686e566/manager/0.log"
Jun 15 06:58:03.206278 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:03.206204    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-nrvh6_8ccda16a-657f-4bca-8e42-e84ce0b5c4a7/s3-init/0.log"
Jun 15 06:58:03.223749 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:03.223726    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-nrvh6_8ccda16a-657f-4bca-8e42-e84ce0b5c4a7/download-hf-model/0.log"
Jun 15 06:58:03.228708 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:03.228682    2572 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-w2k28/perf-node-gather-daemonset-bf7bn"
Jun 15 06:58:03.253925 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:03.253901    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-2wwsz_9c2c0136-4539-4589-b135-969530039e2e/s3-tls-init-custom/0.log"
Jun 15 06:58:03.271690 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:03.271664    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-2wwsz_9c2c0136-4539-4589-b135-969530039e2e/download-hf-model/0.log"
Jun 15 06:58:03.302825 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:03.302782    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-b5gmm_e00644c2-3afe-47da-957b-8bdc8e6f5c24/s3-tls-init-serving/0.log"
Jun 15 06:58:03.320315 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:03.320291    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-b5gmm_e00644c2-3afe-47da-957b-8bdc8e6f5c24/download-hf-model/0.log"
Jun 15 06:58:03.414187 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:03.414158    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-2fpfj_c3e6faac-3bd6-46b7-a063-d693523c800c/seaweedfs-tls-serving/0.log"
Jun 15 06:58:05.673431 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:05.673406    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 06:58:05.709500 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:05.709465    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 06:58:09.093283 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:09.093253    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jwgfl_634277cb-f6c8-4254-b1aa-ccaebfe829a9/kube-multus-additional-cni-plugins/0.log"
Jun 15 06:58:09.121129 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:09.121105    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jwgfl_634277cb-f6c8-4254-b1aa-ccaebfe829a9/egress-router-binary-copy/0.log"
Jun 15 06:58:09.146058 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:09.146027    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jwgfl_634277cb-f6c8-4254-b1aa-ccaebfe829a9/cni-plugins/0.log"
Jun 15 06:58:09.176402 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:09.176373    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jwgfl_634277cb-f6c8-4254-b1aa-ccaebfe829a9/bond-cni-plugin/0.log"
Jun 15 06:58:09.200059 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:09.200034    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jwgfl_634277cb-f6c8-4254-b1aa-ccaebfe829a9/routeoverride-cni/0.log"
Jun 15 06:58:09.226552 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:09.226529    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jwgfl_634277cb-f6c8-4254-b1aa-ccaebfe829a9/whereabouts-cni-bincopy/0.log"
Jun 15 06:58:09.251241 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:09.251219    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-jwgfl_634277cb-f6c8-4254-b1aa-ccaebfe829a9/whereabouts-cni/0.log"
Jun 15 06:58:09.684412 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:09.684382    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-ccxxl_d313c84b-5ccf-445d-bae0-2bc39eea2db2/kube-multus/0.log"
Jun 15 06:58:09.826544 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:09.826515    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vs8ck_57565893-704b-4974-bc5b-c8f9cb932d15/network-metrics-daemon/0.log"
Jun 15 06:58:09.858765 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:09.858737    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vs8ck_57565893-704b-4974-bc5b-c8f9cb932d15/kube-rbac-proxy/0.log"
Jun 15 06:58:11.090316 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:11.090285    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-controller/0.log"
Jun 15 06:58:11.116002 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:11.115971    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/0.log"
Jun 15 06:58:11.116154 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:11.116038    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovn-acl-logging/1.log"
Jun 15 06:58:11.135047 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:11.135022    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/kube-rbac-proxy-node/0.log"
Jun 15 06:58:11.158988 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:11.158953    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/kube-rbac-proxy-ovn-metrics/0.log"
Jun 15 06:58:11.182747 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:11.182723    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/northd/0.log"
Jun 15 06:58:11.205394 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:11.205372    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/nbdb/0.log"
Jun 15 06:58:11.228274 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:11.228255    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/sbdb/0.log"
Jun 15 06:58:11.285940 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:11.285914    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-98qt9_0b5eacff-8b85-4b48-a883-22d29645244a/ovnkube-controller/0.log"
Jun 15 06:58:12.346297 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:12.346268    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-9b7jj_243c1d92-89f7-4ebb-838a-0f18bd3216d8/network-check-target-container/0.log"
Jun 15 06:58:13.440905 ip-10-0-138-16 kubenswrapper[2572]: I0615 06:58:13.440873    2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-lhxv8_0313d892-f05a-4331-b972-c991f2ffe06a/iptables-alerter/0.log"