Jul 08 16:32:12.310367 ip-10-0-128-142 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 08 16:32:12.310383 ip-10-0-128-142 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 08 16:32:12.310393 ip-10-0-128-142 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 08 16:32:12.310736 ip-10-0-128-142 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 08 16:32:22.434943 ip-10-0-128-142 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 08 16:32:22.434958 ip-10-0-128-142 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot c37735e71cb943e8a728b8ab1737d28f --
Jul 08 16:34:23.175976 ip-10-0-128-142 systemd[1]: Starting Kubernetes Kubelet...
Jul 08 16:34:23.552600 ip-10-0-128-142 kubenswrapper[2565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 16:34:23.552600 ip-10-0-128-142 kubenswrapper[2565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 08 16:34:23.552600 ip-10-0-128-142 kubenswrapper[2565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 16:34:23.552600 ip-10-0-128-142 kubenswrapper[2565]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 08 16:34:23.552600 ip-10-0-128-142 kubenswrapper[2565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 08 16:34:23.553316 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.552654    2565 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 08 16:34:23.557128 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557063    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:23.557128 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557088    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:23.557128 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557094    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:23.557128 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557099    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:23.557128 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557104    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:23.557128 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557109    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:23.557128 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557113    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:23.557128 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557118    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:23.557128 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557129    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:23.557128 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557133    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:23.557128 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557138    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:23.557482 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557158    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:23.557482 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557172    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:23.557482 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557278    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:23.557482 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557287    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:23.557482 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557292    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:23.557482 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557296    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:23.557482 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557301    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:23.557482 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557307    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:23.557482 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557311    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:23.557482 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557316    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:23.557482 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557321    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:23.557482 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557325    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:23.557482 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557329    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:23.557482 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557334    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:23.557482 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557339    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:23.557482 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557343    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:23.557482 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557347    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:23.557482 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557351    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:23.557482 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557356    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:23.557482 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557361    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:23.557994 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557365    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:23.557994 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557369    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:23.557994 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557373    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:23.557994 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557377    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:23.557994 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557382    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:23.557994 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557386    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:23.557994 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557390    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:23.557994 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557394    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:23.557994 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557398    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:23.557994 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557403    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:23.557994 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557406    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:23.557994 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557413    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:23.557994 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557417    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:23.557994 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557422    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:23.557994 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557427    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:23.557994 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557431    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:23.557994 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557435    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:23.557994 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557439    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:23.557994 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557443    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:23.557994 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557446    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:23.558474 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557449    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:23.558474 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557452    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:23.558474 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557455    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:23.558474 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557457    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:23.558474 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557461    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:23.558474 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557463    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:23.558474 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557468    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:23.558474 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557472    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:23.558474 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557475    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:23.558474 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557478    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:23.558474 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557481    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:23.558474 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557484    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:23.558474 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557487    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:23.558474 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557490    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:23.558474 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557494    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:23.558474 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557497    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:23.558474 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557500    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:23.558474 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557502    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:23.558474 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557505    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:23.558474 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557507    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:23.558961 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557510    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:23.558961 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557513    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:23.558961 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557515    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:23.558961 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557518    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:23.558961 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557521    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:23.558961 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557525    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:23.558961 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557527    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:23.558961 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557530    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:23.558961 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557533    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:23.558961 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557536    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:23.558961 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557540    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:23.558961 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557543    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:23.558961 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557548    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:23.558961 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557551    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:23.558961 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557553    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:23.558961 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557557    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:23.558961 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557561    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:23.558961 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557565    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:23.559393 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557568    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:23.559393 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557572    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:23.559393 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557574    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:23.559393 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557577    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:23.559393 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557580    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:23.559393 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557583    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:23.559393 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557585    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:23.559393 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557588    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:23.559393 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557590    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:23.559393 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.557593    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:23.559393 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558337    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:23.559393 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558345    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:23.559393 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558349    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:23.559393 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558352    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:23.559393 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558354    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:23.559393 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558357    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:23.559393 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558360    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:23.559393 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558363    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:23.559393 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558365    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:23.559393 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558369    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:23.559917 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558371    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:23.559917 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558374    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:23.559917 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558377    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:23.559917 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558380    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:23.559917 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558382    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:23.559917 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558385    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:23.559917 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558388    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:23.559917 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558391    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:23.559917 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558394    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:23.559917 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558396    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:23.559917 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558399    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:23.559917 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558401    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:23.559917 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558404    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:23.559917 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558407    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:23.559917 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558410    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:23.559917 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558412    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:23.559917 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558415    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:23.559917 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558418    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:23.559917 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558420    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:23.559917 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558423    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:23.560405 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558425    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:23.560405 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558428    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:23.560405 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558431    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:23.560405 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558433    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:23.560405 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558436    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:23.560405 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558440    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:23.560405 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558443    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:23.560405 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558446    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:23.560405 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558448    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:23.560405 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558451    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:23.560405 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558454    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:23.560405 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558457    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:23.560405 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558460    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:23.560405 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558463    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:23.560405 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558466    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:23.560405 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558468    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:23.560405 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558471    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:23.560405 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558473    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:23.560405 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558476    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:23.560886 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558479    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:23.560886 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558482    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:23.560886 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558485    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:23.560886 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558487    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:23.560886 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558490    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:23.560886 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558495    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:23.560886 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558499    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:23.560886 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558502    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:23.560886 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558505    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:23.560886 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558508    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:23.560886 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558511    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:23.560886 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558514    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:23.560886 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558516    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:23.560886 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558519    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:23.560886 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558522    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:23.560886 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558524    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:23.560886 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558527    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:23.560886 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558530    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:23.560886 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558532    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:23.561352 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558535    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:23.561352 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558537    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:23.561352 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558540    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:23.561352 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558543    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:23.561352 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558547    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:23.561352 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558550    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:23.561352 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558553    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:23.561352 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558556    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:23.561352 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558559    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:23.561352 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558562    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:23.561352 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558565    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:23.561352 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558567    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:23.561352 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558570    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:23.561352 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558572    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:23.561352 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558575    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:23.561352 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558578    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:23.561352 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558581    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:23.561352 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558583    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:23.561352 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558586    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:23.561352 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558588    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:23.561852 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558591    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:23.561852 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558593    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:23.561852 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558596    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:23.561852 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558599    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:23.561852 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558602    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:23.561852 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558604    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:23.561852 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558607    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:23.561852 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558609    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:23.561852 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558612    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:23.561852 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558614    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:23.561852 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.558617    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:23.561852 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559011    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:23.561852 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559017    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:23.561852 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559020    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:23.561852 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559023    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:23.561852 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559026    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:23.561852 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559029    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:23.561852 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559031    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:23.561852 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559034    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:23.561852 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559037    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:23.562351 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559039    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:23.562351 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559042    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:23.562351 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559044    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:23.562351 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559047    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:23.562351 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559050    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:23.562351 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559052    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:23.562351 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559055    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:23.562351 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559058    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:23.562351 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559061    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:23.562351 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559064    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:23.562351 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559066    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:23.562351 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559068    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:23.562351 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559071    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:23.562351 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559074    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:23.562351 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559076    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:23.562351 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559079    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:23.562351 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559081    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:23.562351 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559084    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:23.562351 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559086    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:23.562351 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559089    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:23.562847 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559091    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:23.562847 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559093    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:23.562847 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559096    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:23.562847 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559098    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:23.562847 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559101    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:23.562847 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559103    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:23.562847 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559106    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:23.562847 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559109    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:23.562847 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559111    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:23.562847 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559114    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:23.562847 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559117    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:23.562847 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559119    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:23.562847 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559122    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:23.562847 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559124    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:23.562847 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559127    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:23.562847 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559129    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:23.562847 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559132    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:23.562847 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559134    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:23.562847 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559137    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:23.562847 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559140    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:23.563368 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559143    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:23.563368 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559145    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:23.563368 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559148    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:23.563368 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559150    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:23.563368 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559159    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:23.563368 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559161    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:23.563368 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559164    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:23.563368 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559166    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:23.563368 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559169    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:23.563368 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559171    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:23.563368 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559174    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:23.563368 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559176    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:23.563368 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559179    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:23.563368 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559184    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:23.563368 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559188    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:23.563368 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559191    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:23.563368 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559194    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:23.563368 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559197    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:23.563368 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559200    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:23.563837 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559204    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:23.563837 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559206    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:23.563837 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559209    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:23.563837 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559212    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:23.563837 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559214    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:23.563837 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559217    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:23.563837 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559220    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:23.563837 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559223    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:23.563837 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559226    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:23.563837 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559229    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:23.563837 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559231    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:23.563837 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559234    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:23.563837 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559236    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:23.563837 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559239    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:23.563837 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559242    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:23.563837 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559245    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:23.563837 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559247    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:23.563837 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559252    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:23.563837 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559254    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:23.564286 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559257    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:23.564286 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559260    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:23.564286 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559263    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:23.564286 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559265    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:23.564286 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559268    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:23.564286 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559270    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:23.564286 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559273    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:23.564286 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559275    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:23.564286 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559278    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:23.564286 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559280    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:23.564286 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559283    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:23.564286 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.559285    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:23.564286 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559363    2565 flags.go:64] FLAG: --address="0.0.0.0"
Jul 08 16:34:23.564286 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559373    2565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 08 16:34:23.564286 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559382    2565 flags.go:64] FLAG: --anonymous-auth="true"
Jul 08 16:34:23.564286 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559388    2565 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 08 16:34:23.564286 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559394    2565 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 08 16:34:23.564286 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559398    2565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 08 16:34:23.564286 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559403    2565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 08 16:34:23.564286 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559407    2565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 08 16:34:23.564286 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559410    2565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 08 16:34:23.564869 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559414    2565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 08 16:34:23.564869 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559417    2565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 08 16:34:23.564869 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559420    2565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 08 16:34:23.564869 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559423    2565 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 08 16:34:23.564869 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559426    2565 flags.go:64] FLAG: --cgroup-root=""
Jul 08 16:34:23.564869 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559429    2565 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 08 16:34:23.564869 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559432    2565 flags.go:64] FLAG: --client-ca-file=""
Jul 08 16:34:23.564869 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559436    2565 flags.go:64] FLAG: --cloud-provider="external"
Jul 08 16:34:23.564869 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559439    2565 flags.go:64] FLAG: --cluster-dns="[]"
Jul 08 16:34:23.564869 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559446    2565 flags.go:64] FLAG: --cluster-domain=""
Jul 08 16:34:23.564869 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559449    2565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 08 16:34:23.564869 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559451    2565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 08 16:34:23.564869 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559454    2565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 08 16:34:23.564869 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559458    2565 flags.go:64] FLAG: --container-log-max-files="5"
Jul 08 16:34:23.564869 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559461    2565 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 08 16:34:23.564869 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559464    2565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 08 16:34:23.564869 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559467    2565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 08 16:34:23.564869 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559470    2565 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 08 16:34:23.564869 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559473    2565 flags.go:64] FLAG: --contention-profiling="false"
Jul 08 16:34:23.564869 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559476    2565 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 08 16:34:23.564869 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559482    2565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 08 16:34:23.564869 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559485    2565 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 08 16:34:23.564869 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559488    2565 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 08 16:34:23.564869 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559492    2565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 08 16:34:23.565499 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559495    2565 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 08 16:34:23.565499 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559498    2565 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 08 16:34:23.565499 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559501    2565 flags.go:64] FLAG: --enable-load-reader="false"
Jul 08 16:34:23.565499 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559504    2565 flags.go:64] FLAG: --enable-server="true"
Jul 08 16:34:23.565499 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559506    2565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 08 16:34:23.565499 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559511    2565 flags.go:64] FLAG: --event-burst="100"
Jul 08 16:34:23.565499 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559514    2565 flags.go:64] FLAG: --event-qps="50"
Jul 08 16:34:23.565499 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559517    2565 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 08 16:34:23.565499 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559520    2565 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 08 16:34:23.565499 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559522    2565 flags.go:64] FLAG: --eviction-hard=""
Jul 08 16:34:23.565499 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559526    2565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 08 16:34:23.565499 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559529    2565 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 08 16:34:23.565499 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559532    2565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 08 16:34:23.565499 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559535    2565 flags.go:64] FLAG: --eviction-soft=""
Jul 08 16:34:23.565499 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559538    2565 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 08 16:34:23.565499 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559540    2565 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 08 16:34:23.565499 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559544    2565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 08 16:34:23.565499 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559549    2565 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 08 16:34:23.565499 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559552    2565 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 08 16:34:23.565499 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559555    2565 flags.go:64] FLAG: --fail-swap-on="true"
Jul 08 16:34:23.565499 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559557    2565 flags.go:64] FLAG: --feature-gates=""
Jul 08 16:34:23.565499 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559562    2565 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 08 16:34:23.565499 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559565    2565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 08 16:34:23.565499 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559568    2565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 08 16:34:23.565499 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559571    2565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 08 16:34:23.566139 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559574    2565 flags.go:64] FLAG: --healthz-port="10248"
Jul 08 16:34:23.566139 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559577    2565 flags.go:64] FLAG: --help="false"
Jul 08 16:34:23.566139 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559580    2565 flags.go:64] FLAG: --hostname-override="ip-10-0-128-142.ec2.internal"
Jul 08 16:34:23.566139 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559584    2565 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 08 16:34:23.566139 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559587    2565 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 08 16:34:23.566139 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559590    2565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 08 16:34:23.566139 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559594    2565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 08 16:34:23.566139 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559597    2565 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 08 16:34:23.566139 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559600    2565 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 08 16:34:23.566139 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559603    2565 flags.go:64] FLAG: --image-service-endpoint=""
Jul 08 16:34:23.566139 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559606    2565 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 08 16:34:23.566139 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559609    2565 flags.go:64] FLAG: --kube-api-burst="100"
Jul 08 16:34:23.566139 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559611    2565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 08 16:34:23.566139 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559615    2565 flags.go:64] FLAG: --kube-api-qps="50"
Jul 08 16:34:23.566139 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559617    2565 flags.go:64] FLAG: --kube-reserved=""
Jul 08 16:34:23.566139 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559620    2565 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 08 16:34:23.566139 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559623    2565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 08 16:34:23.566139 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559626    2565 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 08 16:34:23.566139 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559629    2565 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 08 16:34:23.566139 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559631    2565 flags.go:64] FLAG: --lock-file=""
Jul 08 16:34:23.566139 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559634    2565 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 08 16:34:23.566139 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559637    2565 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 08 16:34:23.566139 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559640    2565 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 08 16:34:23.566139 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559645    2565 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 08 16:34:23.566729 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559654    2565 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 08 16:34:23.566729 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559657    2565 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 08 16:34:23.566729 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559659    2565 flags.go:64] FLAG: --logging-format="text"
Jul 08 16:34:23.566729 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559662    2565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 08 16:34:23.566729 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559666    2565 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 08 16:34:23.566729 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559668    2565 flags.go:64] FLAG: --manifest-url=""
Jul 08 16:34:23.566729 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559687    2565 flags.go:64] FLAG: --manifest-url-header=""
Jul 08 16:34:23.566729 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559694    2565 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 08 16:34:23.566729 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559699    2565 flags.go:64] FLAG: --max-open-files="1000000"
Jul 08 16:34:23.566729 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559703    2565 flags.go:64] FLAG: --max-pods="110"
Jul 08 16:34:23.566729 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559706    2565 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 08 16:34:23.566729 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559710    2565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 08 16:34:23.566729 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559713    2565 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 08 16:34:23.566729 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559716    2565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 08 16:34:23.566729 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559719    2565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 08 16:34:23.566729 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559722    2565 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 08 16:34:23.566729 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559725    2565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 08 16:34:23.566729 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559732    2565 flags.go:64] FLAG: --node-status-max-images="50"
Jul 08 16:34:23.566729 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559735    2565 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 08 16:34:23.566729 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559738    2565 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 08 16:34:23.566729 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559741    2565 flags.go:64] FLAG: --pod-cidr=""
Jul 08 16:34:23.566729 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559744    2565 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 08 16:34:23.566729 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559749    2565 flags.go:64] FLAG: --pod-manifest-path=""
Jul 08 16:34:23.567289 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559752    2565 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 08 16:34:23.567289 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559755    2565 flags.go:64] FLAG: --pods-per-core="0"
Jul 08 16:34:23.567289 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559758    2565 flags.go:64] FLAG: --port="10250"
Jul 08 16:34:23.567289 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559761    2565 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 08 16:34:23.567289 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559764    2565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-070231ad39c6a14e4"
Jul 08 16:34:23.567289 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559767    2565 flags.go:64] FLAG: --qos-reserved=""
Jul 08 16:34:23.567289 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559770    2565 flags.go:64] FLAG: --read-only-port="10255"
Jul 08 16:34:23.567289 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559773    2565 flags.go:64] FLAG: --register-node="true"
Jul 08 16:34:23.567289 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559776    2565 flags.go:64] FLAG: --register-with-taints=""
Jul 08 16:34:23.567289 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559780    2565 flags.go:64] FLAG: --registry-burst="10"
Jul 08 16:34:23.567289 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559784    2565 flags.go:64] FLAG: --registry-qps="5"
Jul 08 16:34:23.567289 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559787    2565 flags.go:64] FLAG: --reserved-cpus=""
Jul 08 16:34:23.567289 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559790    2565 flags.go:64] FLAG: --reserved-memory=""
Jul 08 16:34:23.567289 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559793    2565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 08 16:34:23.567289 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559797    2565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 08 16:34:23.567289 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559799    2565 flags.go:64] FLAG: --rotate-certificates="false"
Jul 08 16:34:23.567289 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559802    2565 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 08 16:34:23.567289 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559805    2565 flags.go:64] FLAG: --runonce="false"
Jul 08 16:34:23.567289 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559808    2565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 08 16:34:23.567289 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559830    2565 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 08 16:34:23.567289 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559834    2565 flags.go:64] FLAG: --seccomp-default="false"
Jul 08 16:34:23.567289 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559839    2565 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 08 16:34:23.567289 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559842    2565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 08 16:34:23.567289 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559845    2565 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 08 16:34:23.567289 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559848    2565 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 08 16:34:23.567289 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559851    2565 flags.go:64] FLAG: --storage-driver-password="root"
Jul 08 16:34:23.567927 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559854    2565 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 08 16:34:23.567927 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559857    2565 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 08 16:34:23.567927 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559860    2565 flags.go:64] FLAG: --storage-driver-user="root"
Jul 08 16:34:23.567927 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559863    2565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 08 16:34:23.567927 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559866    2565 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 08 16:34:23.567927 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559869    2565 flags.go:64] FLAG: --system-cgroups=""
Jul 08 16:34:23.567927 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559871    2565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 08 16:34:23.567927 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559877    2565 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 08 16:34:23.567927 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559880    2565 flags.go:64] FLAG: --tls-cert-file=""
Jul 08 16:34:23.567927 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559882    2565 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 08 16:34:23.567927 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559887    2565 flags.go:64] FLAG: --tls-min-version=""
Jul 08 16:34:23.567927 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559890    2565 flags.go:64] FLAG: --tls-private-key-file=""
Jul 08 16:34:23.567927 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559892    2565 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 08 16:34:23.567927 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559895    2565 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 08 16:34:23.567927 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559898    2565 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 08 16:34:23.567927 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559902    2565 flags.go:64] FLAG: --v="2"
Jul 08 16:34:23.567927 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559906    2565 flags.go:64] FLAG: --version="false"
Jul 08 16:34:23.567927 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559921    2565 flags.go:64] FLAG: --vmodule=""
Jul 08 16:34:23.567927 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559929    2565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 08 16:34:23.567927 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.559933    2565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 08 16:34:23.567927 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560047    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:23.567927 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560051    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:23.567927 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560054    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:23.567927 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560057    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:23.568541 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560060    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:23.568541 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560063    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:23.568541 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560065    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:23.568541 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560070    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:23.568541 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560072    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:23.568541 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560075    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:23.568541 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560077    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:23.568541 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560080    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:23.568541 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560083    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:23.568541 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560085    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:23.568541 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560088    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:23.568541 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560091    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:23.568541 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560093    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:23.568541 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560096    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:23.568541 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560098    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:23.568541 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560101    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:23.568541 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560103    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:23.568541 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560106    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:23.568541 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560108    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:23.568541 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560111    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:23.569062 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560113    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:23.569062 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560116    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:23.569062 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560118    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:23.569062 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560122    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:23.569062 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560125    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:23.569062 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560127    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:23.569062 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560130    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:23.569062 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560133    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:23.569062 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560137    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:23.569062 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560140    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:23.569062 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560142    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:23.569062 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560145    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:23.569062 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560148    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:23.569062 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560150    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:23.569062 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560153    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:23.569062 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560156    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:23.569062 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560159    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:23.569062 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560161    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:23.569062 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560164    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:23.569529 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560166    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:23.569529 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560169    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:23.569529 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560171    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:23.569529 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560174    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:23.569529 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560177    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:23.569529 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560179    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:23.569529 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560182    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:23.569529 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560185    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:23.569529 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560187    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:23.569529 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560189    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:23.569529 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560192    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:23.569529 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560196    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:23.569529 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560199    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:23.569529 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560202    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:23.569529 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560205    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:23.569529 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560208    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:23.569529 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560212    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:23.569529 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560215    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:23.569529 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560217    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:23.570030 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560220    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:23.570030 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560223    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:23.570030 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560226    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:23.570030 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560228    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:23.570030 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560231    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:23.570030 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560233    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:23.570030 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560236    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:23.570030 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560239    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:23.570030 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560241    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:23.570030 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560245    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:23.570030 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560247    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:23.570030 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560250    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:23.570030 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560252    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:23.570030 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560255    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:23.570030 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560258    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:23.570030 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560260    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:23.570030 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560262    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:23.570030 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560265    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:23.570030 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560268    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:23.570030 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560270    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:23.570530 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560272    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:23.570530 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560275    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:23.570530 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560277    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:23.570530 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560280    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:23.570530 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560283    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:23.570530 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560285    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:23.570530 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560288    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:23.570530 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560290    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:23.570530 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560293    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:23.570530 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560297    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:23.570530 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560299    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:23.570530 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560302    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:23.570530 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560305    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:23.570530 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560307    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:23.570530 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560310    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:23.570530 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560314    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:23.570530 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.560317    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:23.570993 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.560322    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 16:34:23.570993 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.566833    2565 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 08 16:34:23.570993 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.566851    2565 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 08 16:34:23.570993 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.567585    2565 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 08 16:34:23.570993 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.567593    2565 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 08 16:34:23.570993 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567651    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:23.570993 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567656    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:23.570993 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567659    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:23.570993 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567662    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:23.570993 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567665    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:23.570993 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567669    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:23.570993 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567686    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:23.570993 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567689    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:23.570993 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567693    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:23.570993 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567696    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:23.571356 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567699    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:23.571356 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567701    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:23.571356 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567704    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:23.571356 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567706    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:23.571356 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567709    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:23.571356 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567712    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:23.571356 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567715    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:23.571356 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567717    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:23.571356 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567719    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:23.571356 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567722    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:23.571356 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567725    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:23.571356 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567727    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:23.571356 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567730    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:23.571356 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567732    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:23.571356 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567736    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:23.571356 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567739    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:23.571356 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567742    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:23.571356 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567744    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:23.571356 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567747    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:23.571356 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567750    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:23.571860 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567755    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:23.571860 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567759    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:23.571860 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567762    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:23.571860 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567765    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:23.571860 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567768    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:23.571860 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567770    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:23.571860 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567773    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:23.571860 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567775    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:23.571860 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567778    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:23.571860 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567780    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:23.571860 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567783    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:23.571860 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567787    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:23.571860 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567789    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:23.571860 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567792    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:23.571860 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567795    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:23.571860 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567797    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:23.571860 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567800    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:23.571860 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567803    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:23.571860 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567805    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:23.572341 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567808    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:23.572341 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567811    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:23.572341 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567813    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:23.572341 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567816    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:23.572341 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567819    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:23.572341 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567822    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:23.572341 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567824    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:23.572341 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567826    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:23.572341 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567829    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:23.572341 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567832    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:23.572341 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567834    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:23.572341 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567838    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:23.572341 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567841    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:23.572341 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567844    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:23.572341 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567847    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:23.572341 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567849    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:23.572341 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567852    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:23.572341 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567855    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:23.572341 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567858    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:23.572820 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567860    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:23.572820 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567863    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:23.572820 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567865    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:23.572820 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567868    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:23.572820 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567871    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:23.572820 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567873    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:23.572820 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567877    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:23.572820 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567882    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:23.572820 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567884    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:23.572820 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567888    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:23.572820 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567891    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:23.572820 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567894    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:23.572820 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567897    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:23.572820 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567900    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:23.572820 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567903    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:23.572820 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567905    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:23.572820 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567908    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:23.572820 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567910    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:23.572820 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567913    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:23.573284 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567915    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:23.573284 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567918    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:23.573284 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567920    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:23.573284 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567923    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:23.573284 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567925    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:23.573284 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567929    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:23.573284 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567932    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:23.573284 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567935    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:23.573284 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567937    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:23.573284 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567940    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:23.573284 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567943    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:23.573284 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.567946    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:23.573284 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.567951    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 16:34:23.573284 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568062    2565 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 08 16:34:23.573284 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568067    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 08 16:34:23.573666 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568069    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 08 16:34:23.573666 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568072    2565 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 08 16:34:23.573666 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568075    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 08 16:34:23.573666 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568078    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 08 16:34:23.573666 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568081    2565 feature_gate.go:328] unrecognized feature gate: Example
Jul 08 16:34:23.573666 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568084    2565 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 08 16:34:23.573666 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568087    2565 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 08 16:34:23.573666 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568090    2565 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 08 16:34:23.573666 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568093    2565 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 08 16:34:23.573666 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568095    2565 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 08 16:34:23.573666 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568098    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 08 16:34:23.573666 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568100    2565 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 08 16:34:23.573666 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568103    2565 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 08 16:34:23.573666 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568106    2565 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 08 16:34:23.573666 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568108    2565 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 08 16:34:23.573666 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568111    2565 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 08 16:34:23.573666 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568113    2565 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 08 16:34:23.573666 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568116    2565 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 08 16:34:23.573666 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568118    2565 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 08 16:34:23.573666 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568120    2565 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 08 16:34:23.574260 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568123    2565 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 08 16:34:23.574260 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568127    2565 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 08 16:34:23.574260 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568131    2565 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 08 16:34:23.574260 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568134    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 08 16:34:23.574260 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568137    2565 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 08 16:34:23.574260 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568140    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 08 16:34:23.574260 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568143    2565 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 08 16:34:23.574260 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568147    2565 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 08 16:34:23.574260 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568150    2565 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 08 16:34:23.574260 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568153    2565 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 08 16:34:23.574260 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568156    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 08 16:34:23.574260 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568159    2565 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 08 16:34:23.574260 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568161    2565 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 08 16:34:23.574260 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568164    2565 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 08 16:34:23.574260 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568166    2565 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 08 16:34:23.574260 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568169    2565 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 08 16:34:23.574260 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568172    2565 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 08 16:34:23.574260 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568174    2565 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 08 16:34:23.574260 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568177    2565 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 08 16:34:23.574722 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568179    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 08 16:34:23.574722 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568182    2565 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 08 16:34:23.574722 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568184    2565 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 08 16:34:23.574722 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568187    2565 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 08 16:34:23.574722 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568189    2565 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 08 16:34:23.574722 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568192    2565 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 08 16:34:23.574722 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568194    2565 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 08 16:34:23.574722 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568197    2565 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 08 16:34:23.574722 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568199    2565 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 08 16:34:23.574722 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568201    2565 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 08 16:34:23.574722 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568204    2565 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 08 16:34:23.574722 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568206    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 08 16:34:23.574722 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568209    2565 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 08 16:34:23.574722 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568212    2565 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 08 16:34:23.574722 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568214    2565 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 08 16:34:23.574722 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568218    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 08 16:34:23.574722 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568221    2565 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 08 16:34:23.574722 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568223    2565 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 08 16:34:23.574722 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568226    2565 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 08 16:34:23.574722 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568228    2565 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 08 16:34:23.575249 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568231    2565 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 08 16:34:23.575249 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568233    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 08 16:34:23.575249 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568236    2565 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 08 16:34:23.575249 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568238    2565 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 08 16:34:23.575249 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568241    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 08 16:34:23.575249 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568243    2565 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 08 16:34:23.575249 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568246    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 08 16:34:23.575249 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568248    2565 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 08 16:34:23.575249 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568251    2565 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 08 16:34:23.575249 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568253    2565 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 08 16:34:23.575249 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568256    2565 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 08 16:34:23.575249 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568258    2565 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 08 16:34:23.575249 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568260    2565 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 08 16:34:23.575249 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568263    2565 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 08 16:34:23.575249 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568266    2565 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 08 16:34:23.575249 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568268    2565 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 08 16:34:23.575249 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568271    2565 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 08 16:34:23.575249 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568273    2565 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 08 16:34:23.575249 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568276    2565 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 08 16:34:23.575870 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568278    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 08 16:34:23.575870 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568281    2565 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 08 16:34:23.575870 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568283    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 08 16:34:23.575870 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568286    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 08 16:34:23.575870 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568288    2565 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 08 16:34:23.575870 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568291    2565 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 08 16:34:23.575870 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568293    2565 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 08 16:34:23.575870 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568295    2565 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 08 16:34:23.575870 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568299    2565 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 08 16:34:23.575870 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568302    2565 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 08 16:34:23.575870 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568305    2565 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 08 16:34:23.575870 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568309    2565 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 08 16:34:23.575870 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568312    2565 feature_gate.go:328] unrecognized feature gate: Example2
Jul 08 16:34:23.575870 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568314    2565 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 08 16:34:23.575870 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568317    2565 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 08 16:34:23.575870 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568319    2565 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 08 16:34:23.575870 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568321    2565 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 08 16:34:23.575870 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568324    2565 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 08 16:34:23.575870 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:23.568326    2565 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 08 16:34:23.576323 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.568331    2565 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 08 16:34:23.576323 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.568435    2565 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 08 16:34:23.576323 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.570319    2565 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 08 16:34:23.576323 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.571792    2565 server.go:1019] "Starting client certificate rotation"
Jul 08 16:34:23.576323 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.571900    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 08 16:34:23.576323 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.571964    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 08 16:34:23.576323 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.574275    2565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 08 16:34:23.576323 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.575429    2565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 08 16:34:23.585048 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.585030    2565 log.go:25] "Validated CRI v1 runtime API"
Jul 08 16:34:23.587469 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.587450    2565 log.go:25] "Validated CRI v1 image API"
Jul 08 16:34:23.589494 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.589479    2565 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 08 16:34:23.591945 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.591924    2565 fs.go:135] Filesystem UUIDs: map[2ea9a0bb-3a0b-4e9b-842f-d5e56fe3fc7b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b2744633-df25-4e91-93ce-bfca643955ff:/dev/nvme0n1p4]
Jul 08 16:34:23.592002 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.591943    2565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 08 16:34:23.597293 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.597174    2565 manager.go:216] Machine: {Timestamp:2026-07-08 16:34:23.596642033 +0000 UTC m=+0.310003453 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100989 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26c9ece1c2dade73cede5491b91bb4 SystemUUID:ec26c9ec-e1c2-dade-73ce-de5491b91bb4 BootID:c37735e7-1cb9-43e8-a728-b8ab1737d28f Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b2:e2:a7:3e:11 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b2:e2:a7:3e:11 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d2:56:c1:b1:48:e8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 08 16:34:23.597293 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.597286    2565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 08 16:34:23.597475 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.597364    2565 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 08 16:34:23.600525 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.600494    2565 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 08 16:34:23.600667 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.600526    2565 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-142.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 08 16:34:23.601333 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.601322    2565 topology_manager.go:138] "Creating topology manager with none policy"
Jul 08 16:34:23.601367 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.601335    2565 container_manager_linux.go:309] "Creating device plugin manager"
Jul 08 16:34:23.601367 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.601343    2565 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 08 16:34:23.601367 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.601355    2565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 08 16:34:23.602552 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.602541    2565 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 08 16:34:23.605551 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.605537    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 08 16:34:23.605715 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.605704    2565 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 08 16:34:23.605791 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.605736    2565 kubelet.go:486] "Attempting to sync node with API server"
Jul 08 16:34:23.605839 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.605747    2565 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 08 16:34:23.605839 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.605829    2565 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 08 16:34:23.605839 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.605839    2565 kubelet.go:398] "Adding apiserver pod source"
Jul 08 16:34:23.605972 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.605849    2565 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 08 16:34:23.606237 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.606225    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 08 16:34:23.606268 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.606245    2565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 08 16:34:23.609024 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.609006    2565 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 08 16:34:23.609575 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.609551    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 08 16:34:23.610999 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.610981    2565 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 08 16:34:23.611059 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.611016    2565 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 08 16:34:23.612634 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.612622    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 08 16:34:23.612681 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.612638    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 08 16:34:23.612681 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.612645    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 08 16:34:23.612681 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.612651    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 08 16:34:23.612681 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.612660    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 08 16:34:23.612681 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.612666    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 08 16:34:23.612681 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.612674    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 08 16:34:23.612681 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.612680    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 08 16:34:23.612920 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.612686    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 08 16:34:23.612920 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.612692    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 08 16:34:23.612920 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.612701    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 08 16:34:23.612920 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.612707    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 08 16:34:23.613945 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.613931    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 08 16:34:23.613945 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.613941    2565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 08 16:34:23.617289 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.617273    2565 server.go:1268] "Started kubelet"
Jul 08 16:34:23.617406 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.617373    2565 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 08 16:34:23.622162 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.622092    2565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 08 16:34:23.622241 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.622198    2565 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 08 16:34:23.623045 ip-10-0-128-142 systemd[1]: Started Kubernetes Kubelet.
Jul 08 16:34:23.624617 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.624174    2565 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 08 16:34:23.625991 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.625265    2565 server.go:310] "Adding debug handlers to kubelet server"
Jul 08 16:34:23.627459 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:23.627424    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-142.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 08 16:34:23.627556 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.627458    2565 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-142.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 08 16:34:23.627556 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:23.627503    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 08 16:34:23.627672 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:23.627598    2565 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 08 16:34:23.630334 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:23.629236    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-142.ec2.internal.18c05db028b43cba  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-142.ec2.internal,UID:ip-10-0-128-142.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-142.ec2.internal,},FirstTimestamp:2026-07-08 16:34:23.617244346 +0000 UTC m=+0.330605767,LastTimestamp:2026-07-08 16:34:23.617244346 +0000 UTC m=+0.330605767,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-142.ec2.internal,}"
Jul 08 16:34:23.632629 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.632613    2565 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 08 16:34:23.632629 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.632621    2565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 08 16:34:23.633326 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:23.632898    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-142.ec2.internal\" not found"
Jul 08 16:34:23.633725 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.633562    2565 factory.go:55] Registering systemd factory
Jul 08 16:34:23.633725 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.633581    2565 factory.go:223] Registration of the systemd container factory successfully
Jul 08 16:34:23.633725 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.633647    2565 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 08 16:34:23.633725 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.633661    2565 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 08 16:34:23.633934 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.633733    2565 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 08 16:34:23.633934 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.633862    2565 factory.go:153] Registering CRI-O factory
Jul 08 16:34:23.633934 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.633880    2565 factory.go:223] Registration of the crio container factory successfully
Jul 08 16:34:23.634068 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.633952    2565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 08 16:34:23.634068 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.633979    2565 factory.go:103] Registering Raw factory
Jul 08 16:34:23.634068 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.633994    2565 manager.go:1196] Started watching for new ooms in manager
Jul 08 16:34:23.634428 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.634403    2565 reconstruct.go:97] "Volume reconstruction finished"
Jul 08 16:34:23.634428 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.634417    2565 reconciler.go:29] "Reconciler: start to sync state"
Jul 08 16:34:23.634677 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.634663    2565 manager.go:349] Starting recovery of all containers
Jul 08 16:34:23.636004 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:23.635974    2565 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 08 16:34:23.636004 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:23.635973    2565 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-142.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 08 16:34:23.644118 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.644102    2565 manager.go:354] Recovery completed
Jul 08 16:34:23.645831 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:23.645799    2565 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": readdirent /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Jul 08 16:34:23.649183 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.649169    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:23.653841 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.653804    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:23.653940 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.653857    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:23.653940 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.653874    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-142.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:23.654799 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.654778    2565 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 08 16:34:23.654799 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.654799    2565 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 08 16:34:23.654923 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.654825    2565 state_mem.go:36] "Initialized new in-memory state store"
Jul 08 16:34:23.655297 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.655274    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:23.656509 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.656493    2565 policy_none.go:49] "None policy: Start"
Jul 08 16:34:23.656591 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.656514    2565 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 08 16:34:23.656591 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.656529    2565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 08 16:34:23.656704 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:23.656521    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-142.ec2.internal.18c05db02ae2a5af  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-142.ec2.internal,UID:ip-10-0-128-142.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-142.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-142.ec2.internal,},FirstTimestamp:2026-07-08 16:34:23.653840303 +0000 UTC m=+0.367201725,LastTimestamp:2026-07-08 16:34:23.653840303 +0000 UTC m=+0.367201725,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-142.ec2.internal,}"
Jul 08 16:34:23.656928 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.656900    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:23.657001 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.656931    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:23.657001 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.656944    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-142.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:23.657692 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.657679    2565 policy_none.go:47] "Start"
Jul 08 16:34:23.660265 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.660246    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6fpcc"
Jul 08 16:34:23.670025 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:23.669952    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-142.ec2.internal.18c05db02ae30204  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-142.ec2.internal,UID:ip-10-0-128-142.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-142.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-142.ec2.internal,},FirstTimestamp:2026-07-08 16:34:23.65386394 +0000 UTC m=+0.367225362,LastTimestamp:2026-07-08 16:34:23.65386394 +0000 UTC m=+0.367225362,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-142.ec2.internal,}"
Jul 08 16:34:23.672273 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.672255    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6fpcc"
Jul 08 16:34:23.674484 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:23.674393    2565 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-142.ec2.internal.18c05db02ae343b3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-142.ec2.internal,UID:ip-10-0-128-142.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-142.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-142.ec2.internal,},FirstTimestamp:2026-07-08 16:34:23.653880755 +0000 UTC m=+0.367242180,LastTimestamp:2026-07-08 16:34:23.653880755 +0000 UTC m=+0.367242180,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-142.ec2.internal,}"
Jul 08 16:34:23.693352 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.693327    2565 manager.go:337] "Starting Device Plugin manager"
Jul 08 16:34:23.693458 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:23.693386    2565 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 08 16:34:23.693458 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.693400    2565 server.go:87] "Starting device plugin registration server"
Jul 08 16:34:23.693714 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.693700    2565 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 08 16:34:23.693803 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.693717    2565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 08 16:34:23.693856 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.693831    2565 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 08 16:34:23.693942 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.693925    2565 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 08 16:34:23.693942 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.693939    2565 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 08 16:34:23.694427 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:23.694410    2565 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 08 16:34:23.694508 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:23.694460    2565 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-142.ec2.internal\" not found"
Jul 08 16:34:23.795000 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.794946    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:23.796116 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.796038    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:23.796116 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.796070    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:23.796116 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.796082    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-142.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:23.796116 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.796111    2565 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-128-142.ec2.internal"
Jul 08 16:34:23.806899 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.806837    2565 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-128-142.ec2.internal"
Jul 08 16:34:23.806899 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:23.806862    2565 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-142.ec2.internal\": node \"ip-10-0-128-142.ec2.internal\" not found"
Jul 08 16:34:23.814117 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.814081    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 08 16:34:23.815337 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.815321    2565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 08 16:34:23.815437 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.815375    2565 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 08 16:34:23.815437 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.815402    2565 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 08 16:34:23.815546 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:23.815444    2565 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 08 16:34:23.818045 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.818019    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:34:23.822361 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:23.822342    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-142.ec2.internal\" not found"
Jul 08 16:34:23.917780 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.917725    2565 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-142.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-142.ec2.internal"]
Jul 08 16:34:23.917949 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.917831    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:23.918921 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.918901    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:23.919055 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.918932    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:23.919055 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.918942    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-142.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:23.920294 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.920281    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:23.920434 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.920419    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-142.ec2.internal"
Jul 08 16:34:23.920472 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.920453    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:23.921094 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.921077    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:23.921217 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.921112    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:23.921217 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.921126    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-142.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:23.921217 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.921132    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:23.921217 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.921152    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:23.921217 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.921161    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-142.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:23.922529 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.922512    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-142.ec2.internal"
Jul 08 16:34:23.922582 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.922542    2565 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 08 16:34:23.922582 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:23.922545    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-142.ec2.internal\" not found"
Jul 08 16:34:23.923233 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.923215    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-142.ec2.internal" event="NodeHasSufficientMemory"
Jul 08 16:34:23.923332 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.923243    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-142.ec2.internal" event="NodeHasNoDiskPressure"
Jul 08 16:34:23.923332 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.923254    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-142.ec2.internal" event="NodeHasSufficientPID"
Jul 08 16:34:23.936082 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.936066    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8217407d855ec6a5ce57220657163aed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-142.ec2.internal\" (UID: \"8217407d855ec6a5ce57220657163aed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-142.ec2.internal"
Jul 08 16:34:23.936167 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.936091    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8217407d855ec6a5ce57220657163aed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-142.ec2.internal\" (UID: \"8217407d855ec6a5ce57220657163aed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-142.ec2.internal"
Jul 08 16:34:23.936167 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:23.936115    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6c8c7615c18634239b1c0970c2d029c0-config\") pod \"kube-apiserver-proxy-ip-10-0-128-142.ec2.internal\" (UID: \"6c8c7615c18634239b1c0970c2d029c0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-142.ec2.internal"
Jul 08 16:34:23.947509 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:23.947489    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-142.ec2.internal\" not found" node="ip-10-0-128-142.ec2.internal"
Jul 08 16:34:23.952136 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:23.952120    2565 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-142.ec2.internal\" not found" node="ip-10-0-128-142.ec2.internal"
Jul 08 16:34:24.023851 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:24.023819    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-142.ec2.internal\" not found"
Jul 08 16:34:24.037086 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:24.037062    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8217407d855ec6a5ce57220657163aed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-142.ec2.internal\" (UID: \"8217407d855ec6a5ce57220657163aed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-142.ec2.internal"
Jul 08 16:34:24.037181 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:24.037095    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8217407d855ec6a5ce57220657163aed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-142.ec2.internal\" (UID: \"8217407d855ec6a5ce57220657163aed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-142.ec2.internal"
Jul 08 16:34:24.037181 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:24.037114    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6c8c7615c18634239b1c0970c2d029c0-config\") pod \"kube-apiserver-proxy-ip-10-0-128-142.ec2.internal\" (UID: \"6c8c7615c18634239b1c0970c2d029c0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-142.ec2.internal"
Jul 08 16:34:24.037181 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:24.037162    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8217407d855ec6a5ce57220657163aed-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-142.ec2.internal\" (UID: \"8217407d855ec6a5ce57220657163aed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-142.ec2.internal"
Jul 08 16:34:24.037333 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:24.037189    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6c8c7615c18634239b1c0970c2d029c0-config\") pod \"kube-apiserver-proxy-ip-10-0-128-142.ec2.internal\" (UID: \"6c8c7615c18634239b1c0970c2d029c0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-142.ec2.internal"
Jul 08 16:34:24.037333 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:24.037164    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8217407d855ec6a5ce57220657163aed-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-142.ec2.internal\" (UID: \"8217407d855ec6a5ce57220657163aed\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-142.ec2.internal"
Jul 08 16:34:24.125045 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:24.124962    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-142.ec2.internal\" not found"
Jul 08 16:34:24.231444 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:24.231401    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-142.ec2.internal\" not found"
Jul 08 16:34:24.250218 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:24.250189    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-142.ec2.internal"
Jul 08 16:34:24.255596 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:24.255575    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-142.ec2.internal"
Jul 08 16:34:24.333515 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:24.333473    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-142.ec2.internal\" not found"
Jul 08 16:34:24.434344 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:24.434306    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-142.ec2.internal\" not found"
Jul 08 16:34:24.534396 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:24.534373    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-142.ec2.internal\" not found"
Jul 08 16:34:24.571293 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:24.571268    2565 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 08 16:34:24.571811 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:24.571423    2565 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 08 16:34:24.634645 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:24.634613    2565 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-128-142.ec2.internal\" not found"
Jul 08 16:34:24.639545 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:24.639525    2565 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:34:24.679168 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:24.679126    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 16:29:24 +0000 UTC" deadline="2028-02-14 23:19:52.211810508 +0000 UTC"
Jul 08 16:34:24.679168 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:24.679164    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14070h45m27.532651115s"
Jul 08 16:34:24.733893 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:24.733826    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-142.ec2.internal"
Jul 08 16:34:24.750068 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:24.750042    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 08 16:34:24.752035 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:24.752017    2565 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-142.ec2.internal"
Jul 08 16:34:24.763076 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:24.763055    2565 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 08 16:34:24.792954 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:24.792931    2565 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:34:25.003285 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:25.003248    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c8c7615c18634239b1c0970c2d029c0.slice/crio-374e699c5d648fe57e77e84939b84bce02c2eb19ff2a8b2bc2621cb5b40fdf34 WatchSource:0}: Error finding container 374e699c5d648fe57e77e84939b84bce02c2eb19ff2a8b2bc2621cb5b40fdf34: Status 404 returned error can't find the container with id 374e699c5d648fe57e77e84939b84bce02c2eb19ff2a8b2bc2621cb5b40fdf34
Jul 08 16:34:25.008196 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.008177    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 16:34:25.076537 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:25.076509    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8217407d855ec6a5ce57220657163aed.slice/crio-20b58c5915915611bcdaae69b1805e129b55599f3240f9124a3d6ca1832269a3 WatchSource:0}: Error finding container 20b58c5915915611bcdaae69b1805e129b55599f3240f9124a3d6ca1832269a3: Status 404 returned error can't find the container with id 20b58c5915915611bcdaae69b1805e129b55599f3240f9124a3d6ca1832269a3
Jul 08 16:34:25.090284 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.090266    2565 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:34:25.487966 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.487929    2565 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 08 16:34:25.608748 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.608699    2565 apiserver.go:52] "Watching apiserver"
Jul 08 16:34:25.615897 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.615862    2565 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 08 16:34:25.617294 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.617259    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-gb8xd","kube-system/konnectivity-agent-6qgmf","kube-system/kube-apiserver-proxy-ip-10-0-128-142.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pnv7","openshift-dns/node-resolver-hq9qt","openshift-multus/multus-t8qnb","openshift-multus/network-metrics-daemon-bxjqc","kube-system/global-pull-secret-syncer-sxzrc","openshift-cluster-node-tuning-operator/tuned-w9jwf","openshift-image-registry/node-ca-lscjn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-142.ec2.internal","openshift-multus/multus-additional-cni-plugins-97xhl","openshift-network-diagnostics/network-check-target-8m52j","openshift-network-operator/iptables-alerter-hbbx2"]
Jul 08 16:34:25.619373 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.619349    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:34:25.619512 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:25.619450    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxjqc" podUID="d14d465c-2bea-496d-8112-450e25ae7717"
Jul 08 16:34:25.620492 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.620462    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:34:25.620618 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:25.620543    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m52j" podUID="dc585660-f572-4e9e-80b3-63e57d32f369"
Jul 08 16:34:25.622455 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.622432    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hbbx2"
Jul 08 16:34:25.624145 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.623880    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pnv7"
Jul 08 16:34:25.625437 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.625201    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hq9qt"
Jul 08 16:34:25.626118 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.626097    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 08 16:34:25.626765 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.626503    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 08 16:34:25.626860 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.626771    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wmkgx\""
Jul 08 16:34:25.627079 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.627052    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 08 16:34:25.627339 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.627308    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.628006 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.627985    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.628583 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.628558    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 08 16:34:25.628583 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.628573    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rcx72\""
Jul 08 16:34:25.629085 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.629066    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 08 16:34:25.629299 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.629258    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 08 16:34:25.629519 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.629501    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 08 16:34:25.629615 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.629595    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r75cz\""
Jul 08 16:34:25.629745 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.629510    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 08 16:34:25.629810 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.629794    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.630862 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.630841    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6qgmf"
Jul 08 16:34:25.631417 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.631397    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 08 16:34:25.631520 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.631406    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 08 16:34:25.631626 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.631607    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-f2wl6\""
Jul 08 16:34:25.631703 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.631668    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 08 16:34:25.641208 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.641180    2565 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 08 16:34:25.642975 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.642726    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 08 16:34:25.644014 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.643168    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 08 16:34:25.644014 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.643451    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 08 16:34:25.644014 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.643783    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 08 16:34:25.644014 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.643811    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 08 16:34:25.644014 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.643907    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 08 16:34:25.644300 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.644125    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 08 16:34:25.644300 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.644235    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 08 16:34:25.644300 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.644280    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lscjn"
Jul 08 16:34:25.644496 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.644475    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 08 16:34:25.644769 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.644741    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 08 16:34:25.645400 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.644964    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 08 16:34:25.645400 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.645112    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9tkdh\""
Jul 08 16:34:25.645400 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.645218    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m7czv\""
Jul 08 16:34:25.645675 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.645657    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:25.645805 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:25.645780    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxzrc" podUID="3cf84d3d-0028-4d7f-b178-c4a0d8391cdc"
Jul 08 16:34:25.646470 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.646281    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-979db\""
Jul 08 16:34:25.647567 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.647546    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-97xhl"
Jul 08 16:34:25.648012 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.647988    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-os-release\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.648107 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.648044    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-multus-daemon-config\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.648107 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.648090    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-lib-modules\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.648235 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.648123    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-host-cni-netd\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.648235 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.648155    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/42d7b641-e70b-4a9d-b7ee-2acab63c9942-agent-certs\") pod \"konnectivity-agent-6qgmf\" (UID: \"42d7b641-e70b-4a9d-b7ee-2acab63c9942\") " pod="kube-system/konnectivity-agent-6qgmf"
Jul 08 16:34:25.648235 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.648196    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2shks\" (UniqueName: \"kubernetes.io/projected/1d51073c-aced-4933-a4f7-0363e8020735-kube-api-access-2shks\") pod \"node-resolver-hq9qt\" (UID: \"1d51073c-aced-4933-a4f7-0363e8020735\") " pod="openshift-dns/node-resolver-hq9qt"
Jul 08 16:34:25.648389 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.648238    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-etc-kubernetes\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.648389 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.648310    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-host-cni-bin\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.648389 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.648342    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/15283bbe-ca5b-4f0e-a8fb-03f056b9f0b5-iptables-alerter-script\") pod \"iptables-alerter-hbbx2\" (UID: \"15283bbe-ca5b-4f0e-a8fb-03f056b9f0b5\") " pod="openshift-network-operator/iptables-alerter-hbbx2"
Jul 08 16:34:25.648389 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.648377    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-run-ovn\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.648599 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.648410    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/190a9759-19d2-4c1d-abf8-c1197ca4e926-device-dir\") pod \"aws-ebs-csi-driver-node-2pnv7\" (UID: \"190a9759-19d2-4c1d-abf8-c1197ca4e926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pnv7"
Jul 08 16:34:25.648599 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.648439    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15283bbe-ca5b-4f0e-a8fb-03f056b9f0b5-host-slash\") pod \"iptables-alerter-hbbx2\" (UID: \"15283bbe-ca5b-4f0e-a8fb-03f056b9f0b5\") " pod="openshift-network-operator/iptables-alerter-hbbx2"
Jul 08 16:34:25.648599 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.648508    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-var-lib-openvswitch\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.648599 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.648557    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.648599 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.648587    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-multus-cni-dir\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.648902 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.648616    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-cnibin\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.648902 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.648651    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-host-run-netns\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.648902 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.648682    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-host-run-multus-certs\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.648902 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.648753    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-var-lib-kubelet\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.648902 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.648786    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/27f67aff-be14-4330-87d6-56c5059cd8f8-etc-tuned\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.648902 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.648817    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhhdz\" (UniqueName: \"kubernetes.io/projected/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-kube-api-access-rhhdz\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.648902 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.648847    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-host\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.649248 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.648914    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgpmp\" (UniqueName: \"kubernetes.io/projected/27f67aff-be14-4330-87d6-56c5059cd8f8-kube-api-access-fgpmp\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.649248 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.648959    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-host-kubelet\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.649248 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649008    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aea71a23-3932-4b4d-8a12-57ef7d976049-ovn-node-metrics-cert\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.649248 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649066    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwm8p\" (UniqueName: \"kubernetes.io/projected/190a9759-19d2-4c1d-abf8-c1197ca4e926-kube-api-access-bwm8p\") pod \"aws-ebs-csi-driver-node-2pnv7\" (UID: \"190a9759-19d2-4c1d-abf8-c1197ca4e926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pnv7"
Jul 08 16:34:25.649248 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649099    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6f378482-a771-45ff-85cb-3d9b4b887806-serviceca\") pod \"node-ca-lscjn\" (UID: \"6f378482-a771-45ff-85cb-3d9b4b887806\") " pod="openshift-image-registry/node-ca-lscjn"
Jul 08 16:34:25.649248 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649132    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1d51073c-aced-4933-a4f7-0363e8020735-hosts-file\") pod \"node-resolver-hq9qt\" (UID: \"1d51073c-aced-4933-a4f7-0363e8020735\") " pod="openshift-dns/node-resolver-hq9qt"
Jul 08 16:34:25.649248 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649163    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-host-var-lib-kubelet\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.649248 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649193    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-run\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.649248 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649234    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/190a9759-19d2-4c1d-abf8-c1197ca4e926-etc-selinux\") pod \"aws-ebs-csi-driver-node-2pnv7\" (UID: \"190a9759-19d2-4c1d-abf8-c1197ca4e926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pnv7"
Jul 08 16:34:25.649696 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649266    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-host-var-lib-cni-bin\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.649696 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649298    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-etc-sysconfig\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.649696 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649334    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-etc-sysctl-conf\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.649696 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649367    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aea71a23-3932-4b4d-8a12-57ef7d976049-env-overrides\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.649696 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649393    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx4sj\" (UniqueName: \"kubernetes.io/projected/6f378482-a771-45ff-85cb-3d9b4b887806-kube-api-access-zx4sj\") pod \"node-ca-lscjn\" (UID: \"6f378482-a771-45ff-85cb-3d9b4b887806\") " pod="openshift-image-registry/node-ca-lscjn"
Jul 08 16:34:25.649696 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649429    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14d465c-2bea-496d-8112-450e25ae7717-metrics-certs\") pod \"network-metrics-daemon-bxjqc\" (UID: \"d14d465c-2bea-496d-8112-450e25ae7717\") " pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:34:25.649696 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649468    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-hostroot\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.649696 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649499    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-multus-conf-dir\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.649696 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649532    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27f67aff-be14-4330-87d6-56c5059cd8f8-tmp\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.649696 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649559    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-run-systemd\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.649696 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649592    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f378482-a771-45ff-85cb-3d9b4b887806-host\") pod \"node-ca-lscjn\" (UID: \"6f378482-a771-45ff-85cb-3d9b4b887806\") " pod="openshift-image-registry/node-ca-lscjn"
Jul 08 16:34:25.649696 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649622    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-etc-openvswitch\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.649696 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649651    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aea71a23-3932-4b4d-8a12-57ef7d976049-ovnkube-script-lib\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.649696 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649683    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/42d7b641-e70b-4a9d-b7ee-2acab63c9942-konnectivity-ca\") pod \"konnectivity-agent-6qgmf\" (UID: \"42d7b641-e70b-4a9d-b7ee-2acab63c9942\") " pod="kube-system/konnectivity-agent-6qgmf"
Jul 08 16:34:25.650434 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649709    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-cni-binary-copy\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.650434 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649766    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-host-run-k8s-cni-cncf-io\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.650434 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649799    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-etc-sysctl-d\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.650434 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649831    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-systemd-units\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.650434 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649863    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/190a9759-19d2-4c1d-abf8-c1197ca4e926-sys-fs\") pod \"aws-ebs-csi-driver-node-2pnv7\" (UID: \"190a9759-19d2-4c1d-abf8-c1197ca4e926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pnv7"
Jul 08 16:34:25.650434 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649893    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-etc-systemd\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.650434 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649925    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-run-openvswitch\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.650434 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649953    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rptj\" (UniqueName: \"kubernetes.io/projected/15283bbe-ca5b-4f0e-a8fb-03f056b9f0b5-kube-api-access-8rptj\") pod \"iptables-alerter-hbbx2\" (UID: \"15283bbe-ca5b-4f0e-a8fb-03f056b9f0b5\") " pod="openshift-network-operator/iptables-alerter-hbbx2"
Jul 08 16:34:25.650434 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.649984    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/190a9759-19d2-4c1d-abf8-c1197ca4e926-socket-dir\") pod \"aws-ebs-csi-driver-node-2pnv7\" (UID: \"190a9759-19d2-4c1d-abf8-c1197ca4e926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pnv7"
Jul 08 16:34:25.653628 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.653544    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-host-run-netns\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.653628 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.653579    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-host-run-ovn-kubernetes\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.653628 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.653609    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5zv5\" (UniqueName: \"kubernetes.io/projected/dc585660-f572-4e9e-80b3-63e57d32f369-kube-api-access-w5zv5\") pod \"network-check-target-8m52j\" (UID: \"dc585660-f572-4e9e-80b3-63e57d32f369\") " pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:34:25.653830 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.653645    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d51073c-aced-4933-a4f7-0363e8020735-tmp-dir\") pod \"node-resolver-hq9qt\" (UID: \"1d51073c-aced-4933-a4f7-0363e8020735\") " pod="openshift-dns/node-resolver-hq9qt"
Jul 08 16:34:25.653830 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.653672    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-host-var-lib-cni-multus\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.655294 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.655157    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/190a9759-19d2-4c1d-abf8-c1197ca4e926-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2pnv7\" (UID: \"190a9759-19d2-4c1d-abf8-c1197ca4e926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pnv7"
Jul 08 16:34:25.655294 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.655198    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/190a9759-19d2-4c1d-abf8-c1197ca4e926-registration-dir\") pod \"aws-ebs-csi-driver-node-2pnv7\" (UID: \"190a9759-19d2-4c1d-abf8-c1197ca4e926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pnv7"
Jul 08 16:34:25.655294 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.655229    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-node-log\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.655294 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.655269    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-sys\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.655538 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.655304    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-host-slash\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.655538 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.655336    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlc9x\" (UniqueName: \"kubernetes.io/projected/d14d465c-2bea-496d-8112-450e25ae7717-kube-api-access-nlc9x\") pod \"network-metrics-daemon-bxjqc\" (UID: \"d14d465c-2bea-496d-8112-450e25ae7717\") " pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:34:25.655538 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.655383    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-system-cni-dir\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.655538 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.655408    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-multus-socket-dir-parent\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.655538 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.655441    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-etc-kubernetes\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.655538 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.655464    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-etc-modprobe-d\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.655538 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.655487    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aea71a23-3932-4b4d-8a12-57ef7d976049-ovnkube-config\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.655538 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.655512    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkqzc\" (UniqueName: \"kubernetes.io/projected/aea71a23-3932-4b4d-8a12-57ef7d976049-kube-api-access-pkqzc\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.656105 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.655536    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-log-socket\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.666415 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.665550    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 08 16:34:25.666415 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.665649    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cbrbd\""
Jul 08 16:34:25.666415 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.665557    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 08 16:34:25.666415 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.666016    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 08 16:34:25.666415 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.666049    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 08 16:34:25.666415 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.666262    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 08 16:34:25.666415 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.666349    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jx6j6\""
Jul 08 16:34:25.673255 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.673218    2565 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 08 16:34:25.735611 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.735579    2565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 08 16:34:25.755916 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.755820    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-etc-systemd\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.755916 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.755869    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-run-openvswitch\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.755916 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.755906    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rptj\" (UniqueName: \"kubernetes.io/projected/15283bbe-ca5b-4f0e-a8fb-03f056b9f0b5-kube-api-access-8rptj\") pod \"iptables-alerter-hbbx2\" (UID: \"15283bbe-ca5b-4f0e-a8fb-03f056b9f0b5\") " pod="openshift-network-operator/iptables-alerter-hbbx2"
Jul 08 16:34:25.756195 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.755988    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-run-openvswitch\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.756195 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.755991    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-etc-systemd\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.756195 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756019    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/190a9759-19d2-4c1d-abf8-c1197ca4e926-socket-dir\") pod \"aws-ebs-csi-driver-node-2pnv7\" (UID: \"190a9759-19d2-4c1d-abf8-c1197ca4e926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pnv7"
Jul 08 16:34:25.756195 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756065    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn6t5\" (UniqueName: \"kubernetes.io/projected/8086c91a-48e3-4158-a6ba-004f9046fa09-kube-api-access-sn6t5\") pod \"multus-additional-cni-plugins-97xhl\" (UID: \"8086c91a-48e3-4158-a6ba-004f9046fa09\") " pod="openshift-multus/multus-additional-cni-plugins-97xhl"
Jul 08 16:34:25.756195 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756096    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-host-run-netns\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.756195 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756123    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-host-run-ovn-kubernetes\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.756195 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756158    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5zv5\" (UniqueName: \"kubernetes.io/projected/dc585660-f572-4e9e-80b3-63e57d32f369-kube-api-access-w5zv5\") pod \"network-check-target-8m52j\" (UID: \"dc585660-f572-4e9e-80b3-63e57d32f369\") " pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:34:25.756195 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756188    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d51073c-aced-4933-a4f7-0363e8020735-tmp-dir\") pod \"node-resolver-hq9qt\" (UID: \"1d51073c-aced-4933-a4f7-0363e8020735\") " pod="openshift-dns/node-resolver-hq9qt"
Jul 08 16:34:25.756593 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756218    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-host-var-lib-cni-multus\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.756593 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756244    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/190a9759-19d2-4c1d-abf8-c1197ca4e926-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2pnv7\" (UID: \"190a9759-19d2-4c1d-abf8-c1197ca4e926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pnv7"
Jul 08 16:34:25.756593 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756252    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-host-run-ovn-kubernetes\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.756593 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756268    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/190a9759-19d2-4c1d-abf8-c1197ca4e926-registration-dir\") pod \"aws-ebs-csi-driver-node-2pnv7\" (UID: \"190a9759-19d2-4c1d-abf8-c1197ca4e926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pnv7"
Jul 08 16:34:25.756593 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756289    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-host-run-netns\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.756593 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756293    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8086c91a-48e3-4158-a6ba-004f9046fa09-system-cni-dir\") pod \"multus-additional-cni-plugins-97xhl\" (UID: \"8086c91a-48e3-4158-a6ba-004f9046fa09\") " pod="openshift-multus/multus-additional-cni-plugins-97xhl"
Jul 08 16:34:25.756593 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756316    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-node-log\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.756593 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756337    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-sys\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.756593 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756358    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-host-slash\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.756593 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756397    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-dbus\") pod \"global-pull-secret-syncer-sxzrc\" (UID: \"3cf84d3d-0028-4d7f-b178-c4a0d8391cdc\") " pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:25.756593 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756426    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlc9x\" (UniqueName: \"kubernetes.io/projected/d14d465c-2bea-496d-8112-450e25ae7717-kube-api-access-nlc9x\") pod \"network-metrics-daemon-bxjqc\" (UID: \"d14d465c-2bea-496d-8112-450e25ae7717\") " pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:34:25.756593 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756451    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-system-cni-dir\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.756593 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756480    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-multus-socket-dir-parent\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.756593 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756505    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-etc-kubernetes\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.756593 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756530    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-etc-modprobe-d\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.756593 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756556    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aea71a23-3932-4b4d-8a12-57ef7d976049-ovnkube-config\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.756593 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756561    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-node-log\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.757468 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756581    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkqzc\" (UniqueName: \"kubernetes.io/projected/aea71a23-3932-4b4d-8a12-57ef7d976049-kube-api-access-pkqzc\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.757468 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756563    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d51073c-aced-4933-a4f7-0363e8020735-tmp-dir\") pod \"node-resolver-hq9qt\" (UID: \"1d51073c-aced-4933-a4f7-0363e8020735\") " pod="openshift-dns/node-resolver-hq9qt"
Jul 08 16:34:25.757468 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756217    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/190a9759-19d2-4c1d-abf8-c1197ca4e926-socket-dir\") pod \"aws-ebs-csi-driver-node-2pnv7\" (UID: \"190a9759-19d2-4c1d-abf8-c1197ca4e926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pnv7"
Jul 08 16:34:25.757468 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756612    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-host-slash\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.757468 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756630    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-etc-kubernetes\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.757468 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756631    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-log-socket\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.757468 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756658    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/190a9759-19d2-4c1d-abf8-c1197ca4e926-kubelet-dir\") pod \"aws-ebs-csi-driver-node-2pnv7\" (UID: \"190a9759-19d2-4c1d-abf8-c1197ca4e926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pnv7"
Jul 08 16:34:25.757468 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756682    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-sys\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.757468 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756708    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-log-socket\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.757468 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756742    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/190a9759-19d2-4c1d-abf8-c1197ca4e926-registration-dir\") pod \"aws-ebs-csi-driver-node-2pnv7\" (UID: \"190a9759-19d2-4c1d-abf8-c1197ca4e926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pnv7"
Jul 08 16:34:25.757468 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756754    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-host-var-lib-cni-multus\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.757468 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756731    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8086c91a-48e3-4158-a6ba-004f9046fa09-tuning-conf-dir\") pod \"multus-additional-cni-plugins-97xhl\" (UID: \"8086c91a-48e3-4158-a6ba-004f9046fa09\") " pod="openshift-multus/multus-additional-cni-plugins-97xhl"
Jul 08 16:34:25.757468 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756770    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-etc-modprobe-d\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.757468 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756788    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-multus-socket-dir-parent\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.757468 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756795    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-os-release\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.757468 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756804    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-system-cni-dir\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.757468 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756930    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-multus-daemon-config\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.757468 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756948    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-os-release\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.758441 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.756983    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-lib-modules\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.758441 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757023    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-host-cni-netd\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.758441 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757069    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/42d7b641-e70b-4a9d-b7ee-2acab63c9942-agent-certs\") pod \"konnectivity-agent-6qgmf\" (UID: \"42d7b641-e70b-4a9d-b7ee-2acab63c9942\") " pod="kube-system/konnectivity-agent-6qgmf"
Jul 08 16:34:25.758441 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757115    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2shks\" (UniqueName: \"kubernetes.io/projected/1d51073c-aced-4933-a4f7-0363e8020735-kube-api-access-2shks\") pod \"node-resolver-hq9qt\" (UID: \"1d51073c-aced-4933-a4f7-0363e8020735\") " pod="openshift-dns/node-resolver-hq9qt"
Jul 08 16:34:25.758441 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757123    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-host-cni-netd\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.758441 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757113    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-lib-modules\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.758441 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757177    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-etc-kubernetes\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.758441 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757220    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-host-cni-bin\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.758441 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757248    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/15283bbe-ca5b-4f0e-a8fb-03f056b9f0b5-iptables-alerter-script\") pod \"iptables-alerter-hbbx2\" (UID: \"15283bbe-ca5b-4f0e-a8fb-03f056b9f0b5\") " pod="openshift-network-operator/iptables-alerter-hbbx2"
Jul 08 16:34:25.758441 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757274    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-run-ovn\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.758441 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757299    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/190a9759-19d2-4c1d-abf8-c1197ca4e926-device-dir\") pod \"aws-ebs-csi-driver-node-2pnv7\" (UID: \"190a9759-19d2-4c1d-abf8-c1197ca4e926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pnv7"
Jul 08 16:34:25.758441 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757330    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8086c91a-48e3-4158-a6ba-004f9046fa09-os-release\") pod \"multus-additional-cni-plugins-97xhl\" (UID: \"8086c91a-48e3-4158-a6ba-004f9046fa09\") " pod="openshift-multus/multus-additional-cni-plugins-97xhl"
Jul 08 16:34:25.758441 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757358    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-host-cni-bin\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.758441 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757377    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15283bbe-ca5b-4f0e-a8fb-03f056b9f0b5-host-slash\") pod \"iptables-alerter-hbbx2\" (UID: \"15283bbe-ca5b-4f0e-a8fb-03f056b9f0b5\") " pod="openshift-network-operator/iptables-alerter-hbbx2"
Jul 08 16:34:25.758441 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757406    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-run-ovn\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.758441 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757410    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aea71a23-3932-4b4d-8a12-57ef7d976049-ovnkube-config\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.758441 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757298    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-etc-kubernetes\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.759306 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757450    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-var-lib-openvswitch\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.759306 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757407    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-var-lib-openvswitch\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.759306 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757470    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/190a9759-19d2-4c1d-abf8-c1197ca4e926-device-dir\") pod \"aws-ebs-csi-driver-node-2pnv7\" (UID: \"190a9759-19d2-4c1d-abf8-c1197ca4e926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pnv7"
Jul 08 16:34:25.759306 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757487    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.759306 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757523    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/15283bbe-ca5b-4f0e-a8fb-03f056b9f0b5-host-slash\") pod \"iptables-alerter-hbbx2\" (UID: \"15283bbe-ca5b-4f0e-a8fb-03f056b9f0b5\") " pod="openshift-network-operator/iptables-alerter-hbbx2"
Jul 08 16:34:25.759306 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757503    2565 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 08 16:34:25.759306 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757545    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-multus-cni-dir\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.759306 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757583    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-multus-daemon-config\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.759306 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757598    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-cnibin\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.759306 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757593    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.759306 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757678    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-cnibin\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.759306 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757716    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-host-run-netns\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.759306 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757761    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-host-run-multus-certs\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.759306 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757764    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-multus-cni-dir\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.759306 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757781    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-host-run-netns\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.759306 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757790    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-var-lib-kubelet\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.759306 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757815    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/27f67aff-be14-4330-87d6-56c5059cd8f8-etc-tuned\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.759306 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757826    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/15283bbe-ca5b-4f0e-a8fb-03f056b9f0b5-iptables-alerter-script\") pod \"iptables-alerter-hbbx2\" (UID: \"15283bbe-ca5b-4f0e-a8fb-03f056b9f0b5\") " pod="openshift-network-operator/iptables-alerter-hbbx2"
Jul 08 16:34:25.760237 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757843    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhhdz\" (UniqueName: \"kubernetes.io/projected/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-kube-api-access-rhhdz\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.760237 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757810    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-host-run-multus-certs\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.760237 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757869    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-host\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.760237 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757859    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-var-lib-kubelet\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.760237 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757896    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgpmp\" (UniqueName: \"kubernetes.io/projected/27f67aff-be14-4330-87d6-56c5059cd8f8-kube-api-access-fgpmp\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.760237 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757922    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-host-kubelet\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.760237 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757943    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-host\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.760237 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757953    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aea71a23-3932-4b4d-8a12-57ef7d976049-ovn-node-metrics-cert\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.760237 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757979    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwm8p\" (UniqueName: \"kubernetes.io/projected/190a9759-19d2-4c1d-abf8-c1197ca4e926-kube-api-access-bwm8p\") pod \"aws-ebs-csi-driver-node-2pnv7\" (UID: \"190a9759-19d2-4c1d-abf8-c1197ca4e926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pnv7"
Jul 08 16:34:25.760237 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.757991    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-host-kubelet\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.760237 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758005    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6f378482-a771-45ff-85cb-3d9b4b887806-serviceca\") pod \"node-ca-lscjn\" (UID: \"6f378482-a771-45ff-85cb-3d9b4b887806\") " pod="openshift-image-registry/node-ca-lscjn"
Jul 08 16:34:25.760237 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758087    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1d51073c-aced-4933-a4f7-0363e8020735-hosts-file\") pod \"node-resolver-hq9qt\" (UID: \"1d51073c-aced-4933-a4f7-0363e8020735\") " pod="openshift-dns/node-resolver-hq9qt"
Jul 08 16:34:25.760237 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758127    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-host-var-lib-kubelet\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.760237 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758155    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-run\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.760237 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758180    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/190a9759-19d2-4c1d-abf8-c1197ca4e926-etc-selinux\") pod \"aws-ebs-csi-driver-node-2pnv7\" (UID: \"190a9759-19d2-4c1d-abf8-c1197ca4e926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pnv7"
Jul 08 16:34:25.760237 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758205    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-host-var-lib-cni-bin\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.760237 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758206    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1d51073c-aced-4933-a4f7-0363e8020735-hosts-file\") pod \"node-resolver-hq9qt\" (UID: \"1d51073c-aced-4933-a4f7-0363e8020735\") " pod="openshift-dns/node-resolver-hq9qt"
Jul 08 16:34:25.760237 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758247    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-etc-sysconfig\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.761190 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758276    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-etc-sysctl-conf\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.761190 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758267    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-host-var-lib-cni-bin\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.761190 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758305    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aea71a23-3932-4b4d-8a12-57ef7d976049-env-overrides\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.761190 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758337    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-host-var-lib-kubelet\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.761190 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758338    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx4sj\" (UniqueName: \"kubernetes.io/projected/6f378482-a771-45ff-85cb-3d9b4b887806-kube-api-access-zx4sj\") pod \"node-ca-lscjn\" (UID: \"6f378482-a771-45ff-85cb-3d9b4b887806\") " pod="openshift-image-registry/node-ca-lscjn"
Jul 08 16:34:25.761190 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758373    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8086c91a-48e3-4158-a6ba-004f9046fa09-cni-binary-copy\") pod \"multus-additional-cni-plugins-97xhl\" (UID: \"8086c91a-48e3-4158-a6ba-004f9046fa09\") " pod="openshift-multus/multus-additional-cni-plugins-97xhl"
Jul 08 16:34:25.761190 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758402    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8086c91a-48e3-4158-a6ba-004f9046fa09-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-97xhl\" (UID: \"8086c91a-48e3-4158-a6ba-004f9046fa09\") " pod="openshift-multus/multus-additional-cni-plugins-97xhl"
Jul 08 16:34:25.761190 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758429    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-original-pull-secret\") pod \"global-pull-secret-syncer-sxzrc\" (UID: \"3cf84d3d-0028-4d7f-b178-c4a0d8391cdc\") " pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:25.761190 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758460    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14d465c-2bea-496d-8112-450e25ae7717-metrics-certs\") pod \"network-metrics-daemon-bxjqc\" (UID: \"d14d465c-2bea-496d-8112-450e25ae7717\") " pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:34:25.761190 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758488    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-hostroot\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.761190 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758277    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-run\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.761190 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758503    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/6f378482-a771-45ff-85cb-3d9b4b887806-serviceca\") pod \"node-ca-lscjn\" (UID: \"6f378482-a771-45ff-85cb-3d9b4b887806\") " pod="openshift-image-registry/node-ca-lscjn"
Jul 08 16:34:25.761190 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758516    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-multus-conf-dir\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.761190 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758529    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-etc-sysconfig\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.761190 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758374    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/190a9759-19d2-4c1d-abf8-c1197ca4e926-etc-selinux\") pod \"aws-ebs-csi-driver-node-2pnv7\" (UID: \"190a9759-19d2-4c1d-abf8-c1197ca4e926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pnv7"
Jul 08 16:34:25.761190 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758554    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27f67aff-be14-4330-87d6-56c5059cd8f8-tmp\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.761190 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758489    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-etc-sysctl-conf\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.762049 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758615    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-hostroot\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.762049 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:25.758632    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:25.762049 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758650    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-multus-conf-dir\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.762049 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758660    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-run-systemd\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.762049 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758688    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f378482-a771-45ff-85cb-3d9b4b887806-host\") pod \"node-ca-lscjn\" (UID: \"6f378482-a771-45ff-85cb-3d9b4b887806\") " pod="openshift-image-registry/node-ca-lscjn"
Jul 08 16:34:25.762049 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:25.758806    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d14d465c-2bea-496d-8112-450e25ae7717-metrics-certs podName:d14d465c-2bea-496d-8112-450e25ae7717 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:26.258768281 +0000 UTC m=+2.972129703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d14d465c-2bea-496d-8112-450e25ae7717-metrics-certs") pod "network-metrics-daemon-bxjqc" (UID: "d14d465c-2bea-496d-8112-450e25ae7717") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:25.762049 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758829    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8086c91a-48e3-4158-a6ba-004f9046fa09-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-97xhl\" (UID: \"8086c91a-48e3-4158-a6ba-004f9046fa09\") " pod="openshift-multus/multus-additional-cni-plugins-97xhl"
Jul 08 16:34:25.762049 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758838    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-run-systemd\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.762049 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758863    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-etc-openvswitch\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.762049 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758915    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6f378482-a771-45ff-85cb-3d9b4b887806-host\") pod \"node-ca-lscjn\" (UID: \"6f378482-a771-45ff-85cb-3d9b4b887806\") " pod="openshift-image-registry/node-ca-lscjn"
Jul 08 16:34:25.762049 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758944    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aea71a23-3932-4b4d-8a12-57ef7d976049-env-overrides\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.762049 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758947    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aea71a23-3932-4b4d-8a12-57ef7d976049-ovnkube-script-lib\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.762049 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.758984    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/42d7b641-e70b-4a9d-b7ee-2acab63c9942-konnectivity-ca\") pod \"konnectivity-agent-6qgmf\" (UID: \"42d7b641-e70b-4a9d-b7ee-2acab63c9942\") " pod="kube-system/konnectivity-agent-6qgmf"
Jul 08 16:34:25.762049 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.759006    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8086c91a-48e3-4158-a6ba-004f9046fa09-cnibin\") pod \"multus-additional-cni-plugins-97xhl\" (UID: \"8086c91a-48e3-4158-a6ba-004f9046fa09\") " pod="openshift-multus/multus-additional-cni-plugins-97xhl"
Jul 08 16:34:25.762049 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.759009    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-etc-openvswitch\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.762049 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.759047    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-kubelet-config\") pod \"global-pull-secret-syncer-sxzrc\" (UID: \"3cf84d3d-0028-4d7f-b178-c4a0d8391cdc\") " pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:25.762049 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.759078    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-cni-binary-copy\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.762961 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.759153    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-host-run-k8s-cni-cncf-io\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.762961 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.759181    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-etc-sysctl-d\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.762961 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.759209    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-systemd-units\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.762961 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.759236    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/190a9759-19d2-4c1d-abf8-c1197ca4e926-sys-fs\") pod \"aws-ebs-csi-driver-node-2pnv7\" (UID: \"190a9759-19d2-4c1d-abf8-c1197ca4e926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pnv7"
Jul 08 16:34:25.762961 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.759338    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aea71a23-3932-4b4d-8a12-57ef7d976049-ovnkube-script-lib\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.762961 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.759377    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-host-run-k8s-cni-cncf-io\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.762961 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.759469    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aea71a23-3932-4b4d-8a12-57ef7d976049-systemd-units\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.762961 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.759530    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/42d7b641-e70b-4a9d-b7ee-2acab63c9942-konnectivity-ca\") pod \"konnectivity-agent-6qgmf\" (UID: \"42d7b641-e70b-4a9d-b7ee-2acab63c9942\") " pod="kube-system/konnectivity-agent-6qgmf"
Jul 08 16:34:25.762961 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.759584    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/27f67aff-be14-4330-87d6-56c5059cd8f8-etc-sysctl-d\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.762961 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.759587    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/190a9759-19d2-4c1d-abf8-c1197ca4e926-sys-fs\") pod \"aws-ebs-csi-driver-node-2pnv7\" (UID: \"190a9759-19d2-4c1d-abf8-c1197ca4e926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pnv7"
Jul 08 16:34:25.762961 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.759745    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-cni-binary-copy\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.762961 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.759857    2565 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mjw2g"
Jul 08 16:34:25.762961 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.761618    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/27f67aff-be14-4330-87d6-56c5059cd8f8-etc-tuned\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.762961 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.762415    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aea71a23-3932-4b4d-8a12-57ef7d976049-ovn-node-metrics-cert\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.762961 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.762586    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/42d7b641-e70b-4a9d-b7ee-2acab63c9942-agent-certs\") pod \"konnectivity-agent-6qgmf\" (UID: \"42d7b641-e70b-4a9d-b7ee-2acab63c9942\") " pod="kube-system/konnectivity-agent-6qgmf"
Jul 08 16:34:25.763716 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.763693    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/27f67aff-be14-4330-87d6-56c5059cd8f8-tmp\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.767858 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.767838    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rptj\" (UniqueName: \"kubernetes.io/projected/15283bbe-ca5b-4f0e-a8fb-03f056b9f0b5-kube-api-access-8rptj\") pod \"iptables-alerter-hbbx2\" (UID: \"15283bbe-ca5b-4f0e-a8fb-03f056b9f0b5\") " pod="openshift-network-operator/iptables-alerter-hbbx2"
Jul 08 16:34:25.784567 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.784250    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlc9x\" (UniqueName: \"kubernetes.io/projected/d14d465c-2bea-496d-8112-450e25ae7717-kube-api-access-nlc9x\") pod \"network-metrics-daemon-bxjqc\" (UID: \"d14d465c-2bea-496d-8112-450e25ae7717\") " pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:34:25.784567 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.784518    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkqzc\" (UniqueName: \"kubernetes.io/projected/aea71a23-3932-4b4d-8a12-57ef7d976049-kube-api-access-pkqzc\") pod \"ovnkube-node-gb8xd\" (UID: \"aea71a23-3932-4b4d-8a12-57ef7d976049\") " pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:25.785382 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:25.784988    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:25.785382 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:25.785017    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:25.785382 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:25.785031    2565 projected.go:196] Error preparing data for projected volume kube-api-access-w5zv5 for pod openshift-network-diagnostics/network-check-target-8m52j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:25.785382 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:25.785180    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc585660-f572-4e9e-80b3-63e57d32f369-kube-api-access-w5zv5 podName:dc585660-f572-4e9e-80b3-63e57d32f369 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:26.285156213 +0000 UTC m=+2.998517647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-w5zv5" (UniqueName: "kubernetes.io/projected/dc585660-f572-4e9e-80b3-63e57d32f369-kube-api-access-w5zv5") pod "network-check-target-8m52j" (UID: "dc585660-f572-4e9e-80b3-63e57d32f369") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:25.787805 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.787780    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2shks\" (UniqueName: \"kubernetes.io/projected/1d51073c-aced-4933-a4f7-0363e8020735-kube-api-access-2shks\") pod \"node-resolver-hq9qt\" (UID: \"1d51073c-aced-4933-a4f7-0363e8020735\") " pod="openshift-dns/node-resolver-hq9qt"
Jul 08 16:34:25.793086 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.793055    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx4sj\" (UniqueName: \"kubernetes.io/projected/6f378482-a771-45ff-85cb-3d9b4b887806-kube-api-access-zx4sj\") pod \"node-ca-lscjn\" (UID: \"6f378482-a771-45ff-85cb-3d9b4b887806\") " pod="openshift-image-registry/node-ca-lscjn"
Jul 08 16:34:25.797174 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.797129    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwm8p\" (UniqueName: \"kubernetes.io/projected/190a9759-19d2-4c1d-abf8-c1197ca4e926-kube-api-access-bwm8p\") pod \"aws-ebs-csi-driver-node-2pnv7\" (UID: \"190a9759-19d2-4c1d-abf8-c1197ca4e926\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pnv7"
Jul 08 16:34:25.798957 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.798926    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgpmp\" (UniqueName: \"kubernetes.io/projected/27f67aff-be14-4330-87d6-56c5059cd8f8-kube-api-access-fgpmp\") pod \"tuned-w9jwf\" (UID: \"27f67aff-be14-4330-87d6-56c5059cd8f8\") " pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.803620 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.803216    2565 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mjw2g"
Jul 08 16:34:25.805182 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.805159    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhhdz\" (UniqueName: \"kubernetes.io/projected/7b9bc9aa-a1bc-4af2-8dff-21213565dd18-kube-api-access-rhhdz\") pod \"multus-t8qnb\" (UID: \"7b9bc9aa-a1bc-4af2-8dff-21213565dd18\") " pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.821741 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.821678    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-142.ec2.internal" event={"ID":"8217407d855ec6a5ce57220657163aed","Type":"ContainerStarted","Data":"20b58c5915915611bcdaae69b1805e129b55599f3240f9124a3d6ca1832269a3"}
Jul 08 16:34:25.823129 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.823101    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-142.ec2.internal" event={"ID":"6c8c7615c18634239b1c0970c2d029c0","Type":"ContainerStarted","Data":"374e699c5d648fe57e77e84939b84bce02c2eb19ff2a8b2bc2621cb5b40fdf34"}
Jul 08 16:34:25.859645 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.859550    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8086c91a-48e3-4158-a6ba-004f9046fa09-system-cni-dir\") pod \"multus-additional-cni-plugins-97xhl\" (UID: \"8086c91a-48e3-4158-a6ba-004f9046fa09\") " pod="openshift-multus/multus-additional-cni-plugins-97xhl"
Jul 08 16:34:25.859822 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.859659    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-dbus\") pod \"global-pull-secret-syncer-sxzrc\" (UID: \"3cf84d3d-0028-4d7f-b178-c4a0d8391cdc\") " pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:25.859822 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.859698    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8086c91a-48e3-4158-a6ba-004f9046fa09-tuning-conf-dir\") pod \"multus-additional-cni-plugins-97xhl\" (UID: \"8086c91a-48e3-4158-a6ba-004f9046fa09\") " pod="openshift-multus/multus-additional-cni-plugins-97xhl"
Jul 08 16:34:25.859822 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.859728    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8086c91a-48e3-4158-a6ba-004f9046fa09-os-release\") pod \"multus-additional-cni-plugins-97xhl\" (UID: \"8086c91a-48e3-4158-a6ba-004f9046fa09\") " pod="openshift-multus/multus-additional-cni-plugins-97xhl"
Jul 08 16:34:25.859822 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.859791    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8086c91a-48e3-4158-a6ba-004f9046fa09-system-cni-dir\") pod \"multus-additional-cni-plugins-97xhl\" (UID: \"8086c91a-48e3-4158-a6ba-004f9046fa09\") " pod="openshift-multus/multus-additional-cni-plugins-97xhl"
Jul 08 16:34:25.859822 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.859819    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8086c91a-48e3-4158-a6ba-004f9046fa09-cni-binary-copy\") pod \"multus-additional-cni-plugins-97xhl\" (UID: \"8086c91a-48e3-4158-a6ba-004f9046fa09\") " pod="openshift-multus/multus-additional-cni-plugins-97xhl"
Jul 08 16:34:25.860106 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.859845    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8086c91a-48e3-4158-a6ba-004f9046fa09-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-97xhl\" (UID: \"8086c91a-48e3-4158-a6ba-004f9046fa09\") " pod="openshift-multus/multus-additional-cni-plugins-97xhl"
Jul 08 16:34:25.860106 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.859873    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-original-pull-secret\") pod \"global-pull-secret-syncer-sxzrc\" (UID: \"3cf84d3d-0028-4d7f-b178-c4a0d8391cdc\") " pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:25.860106 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.859912    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-dbus\") pod \"global-pull-secret-syncer-sxzrc\" (UID: \"3cf84d3d-0028-4d7f-b178-c4a0d8391cdc\") " pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:25.860106 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.859918    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8086c91a-48e3-4158-a6ba-004f9046fa09-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-97xhl\" (UID: \"8086c91a-48e3-4158-a6ba-004f9046fa09\") " pod="openshift-multus/multus-additional-cni-plugins-97xhl"
Jul 08 16:34:25.860106 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.859961    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8086c91a-48e3-4158-a6ba-004f9046fa09-os-release\") pod \"multus-additional-cni-plugins-97xhl\" (UID: \"8086c91a-48e3-4158-a6ba-004f9046fa09\") " pod="openshift-multus/multus-additional-cni-plugins-97xhl"
Jul 08 16:34:25.860106 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.859978    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8086c91a-48e3-4158-a6ba-004f9046fa09-cnibin\") pod \"multus-additional-cni-plugins-97xhl\" (UID: \"8086c91a-48e3-4158-a6ba-004f9046fa09\") " pod="openshift-multus/multus-additional-cni-plugins-97xhl"
Jul 08 16:34:25.860106 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.859992    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8086c91a-48e3-4158-a6ba-004f9046fa09-tuning-conf-dir\") pod \"multus-additional-cni-plugins-97xhl\" (UID: \"8086c91a-48e3-4158-a6ba-004f9046fa09\") " pod="openshift-multus/multus-additional-cni-plugins-97xhl"
Jul 08 16:34:25.860106 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.860007    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-kubelet-config\") pod \"global-pull-secret-syncer-sxzrc\" (UID: \"3cf84d3d-0028-4d7f-b178-c4a0d8391cdc\") " pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:25.860106 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.860054    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn6t5\" (UniqueName: \"kubernetes.io/projected/8086c91a-48e3-4158-a6ba-004f9046fa09-kube-api-access-sn6t5\") pod \"multus-additional-cni-plugins-97xhl\" (UID: \"8086c91a-48e3-4158-a6ba-004f9046fa09\") " pod="openshift-multus/multus-additional-cni-plugins-97xhl"
Jul 08 16:34:25.860106 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:25.860070    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:25.860647 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:25.860530    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-original-pull-secret podName:3cf84d3d-0028-4d7f-b178-c4a0d8391cdc nodeName:}" failed. No retries permitted until 2026-07-08 16:34:26.360498373 +0000 UTC m=+3.073859835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-original-pull-secret") pod "global-pull-secret-syncer-sxzrc" (UID: "3cf84d3d-0028-4d7f-b178-c4a0d8391cdc") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:25.861328 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.861021    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8086c91a-48e3-4158-a6ba-004f9046fa09-cnibin\") pod \"multus-additional-cni-plugins-97xhl\" (UID: \"8086c91a-48e3-4158-a6ba-004f9046fa09\") " pod="openshift-multus/multus-additional-cni-plugins-97xhl"
Jul 08 16:34:25.861328 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.861116    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-kubelet-config\") pod \"global-pull-secret-syncer-sxzrc\" (UID: \"3cf84d3d-0028-4d7f-b178-c4a0d8391cdc\") " pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:25.861521 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.861336    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8086c91a-48e3-4158-a6ba-004f9046fa09-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-97xhl\" (UID: \"8086c91a-48e3-4158-a6ba-004f9046fa09\") " pod="openshift-multus/multus-additional-cni-plugins-97xhl"
Jul 08 16:34:25.861521 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.861408    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8086c91a-48e3-4158-a6ba-004f9046fa09-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-97xhl\" (UID: \"8086c91a-48e3-4158-a6ba-004f9046fa09\") " pod="openshift-multus/multus-additional-cni-plugins-97xhl"
Jul 08 16:34:25.861521 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.861425    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8086c91a-48e3-4158-a6ba-004f9046fa09-cni-binary-copy\") pod \"multus-additional-cni-plugins-97xhl\" (UID: \"8086c91a-48e3-4158-a6ba-004f9046fa09\") " pod="openshift-multus/multus-additional-cni-plugins-97xhl"
Jul 08 16:34:25.874514 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.874486    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn6t5\" (UniqueName: \"kubernetes.io/projected/8086c91a-48e3-4158-a6ba-004f9046fa09-kube-api-access-sn6t5\") pod \"multus-additional-cni-plugins-97xhl\" (UID: \"8086c91a-48e3-4158-a6ba-004f9046fa09\") " pod="openshift-multus/multus-additional-cni-plugins-97xhl"
Jul 08 16:34:25.951276 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.951225    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hbbx2"
Jul 08 16:34:25.958505 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.958474    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pnv7"
Jul 08 16:34:25.970531 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.970507    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hq9qt"
Jul 08 16:34:25.979074 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.979053    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-t8qnb"
Jul 08 16:34:25.989226 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.989139    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w9jwf"
Jul 08 16:34:25.997025 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:25.996996    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:26.005687 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:26.005665    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6qgmf"
Jul 08 16:34:26.013558 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:26.013532    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lscjn"
Jul 08 16:34:26.020943 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:26.020912    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-97xhl"
Jul 08 16:34:26.263602 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:26.263512    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14d465c-2bea-496d-8112-450e25ae7717-metrics-certs\") pod \"network-metrics-daemon-bxjqc\" (UID: \"d14d465c-2bea-496d-8112-450e25ae7717\") " pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:34:26.263776 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:26.263659    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:26.263776 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:26.263744    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d14d465c-2bea-496d-8112-450e25ae7717-metrics-certs podName:d14d465c-2bea-496d-8112-450e25ae7717 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:27.26370778 +0000 UTC m=+3.977069200 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d14d465c-2bea-496d-8112-450e25ae7717-metrics-certs") pod "network-metrics-daemon-bxjqc" (UID: "d14d465c-2bea-496d-8112-450e25ae7717") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:26.364649 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:26.364604    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5zv5\" (UniqueName: \"kubernetes.io/projected/dc585660-f572-4e9e-80b3-63e57d32f369-kube-api-access-w5zv5\") pod \"network-check-target-8m52j\" (UID: \"dc585660-f572-4e9e-80b3-63e57d32f369\") " pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:34:26.364845 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:26.364694    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-original-pull-secret\") pod \"global-pull-secret-syncer-sxzrc\" (UID: \"3cf84d3d-0028-4d7f-b178-c4a0d8391cdc\") " pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:26.364845 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:26.364810    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:26.364845 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:26.364835    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:26.364845 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:26.364845    2565 projected.go:196] Error preparing data for projected volume kube-api-access-w5zv5 for pod openshift-network-diagnostics/network-check-target-8m52j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:26.365041 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:26.364852    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:26.365041 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:26.364907    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc585660-f572-4e9e-80b3-63e57d32f369-kube-api-access-w5zv5 podName:dc585660-f572-4e9e-80b3-63e57d32f369 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:27.364888844 +0000 UTC m=+4.078250273 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-w5zv5" (UniqueName: "kubernetes.io/projected/dc585660-f572-4e9e-80b3-63e57d32f369-kube-api-access-w5zv5") pod "network-check-target-8m52j" (UID: "dc585660-f572-4e9e-80b3-63e57d32f369") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:26.365041 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:26.364928    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-original-pull-secret podName:3cf84d3d-0028-4d7f-b178-c4a0d8391cdc nodeName:}" failed. No retries permitted until 2026-07-08 16:34:27.364920724 +0000 UTC m=+4.078282130 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-original-pull-secret") pod "global-pull-secret-syncer-sxzrc" (UID: "3cf84d3d-0028-4d7f-b178-c4a0d8391cdc") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:26.807038 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:26.806989    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 16:29:25 +0000 UTC" deadline="2027-12-23 07:24:27.622021312 +0000 UTC"
Jul 08 16:34:26.807038 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:26.807036    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12782h50m0.81499151s"
Jul 08 16:34:26.816275 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:26.816244    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:26.816443 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:26.816386    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxzrc" podUID="3cf84d3d-0028-4d7f-b178-c4a0d8391cdc"
Jul 08 16:34:27.003130 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:27.003097    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15283bbe_ca5b_4f0e_a8fb_03f056b9f0b5.slice/crio-822dfae992eccbc223ad8cc18317a1b70b21fd12ddc46d40a0a88e11502a98df WatchSource:0}: Error finding container 822dfae992eccbc223ad8cc18317a1b70b21fd12ddc46d40a0a88e11502a98df: Status 404 returned error can't find the container with id 822dfae992eccbc223ad8cc18317a1b70b21fd12ddc46d40a0a88e11502a98df
Jul 08 16:34:27.004813 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:27.004771    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42d7b641_e70b_4a9d_b7ee_2acab63c9942.slice/crio-36a4f51ebc79b55cdd68968f72b41580933fd367e0dd0569cecde5b4db9fac9f WatchSource:0}: Error finding container 36a4f51ebc79b55cdd68968f72b41580933fd367e0dd0569cecde5b4db9fac9f: Status 404 returned error can't find the container with id 36a4f51ebc79b55cdd68968f72b41580933fd367e0dd0569cecde5b4db9fac9f
Jul 08 16:34:27.006285 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:27.006243    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190a9759_19d2_4c1d_abf8_c1197ca4e926.slice/crio-4351a639bdeeed719a22d99fde583fd628e3eda2d402cdbec2d605be84e7827e WatchSource:0}: Error finding container 4351a639bdeeed719a22d99fde583fd628e3eda2d402cdbec2d605be84e7827e: Status 404 returned error can't find the container with id 4351a639bdeeed719a22d99fde583fd628e3eda2d402cdbec2d605be84e7827e
Jul 08 16:34:27.010385 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:27.010359    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8086c91a_48e3_4158_a6ba_004f9046fa09.slice/crio-810843c5fc297b009bbe9a5daf083d80137bad1b09e196c6e2f9c1d08250d8b6 WatchSource:0}: Error finding container 810843c5fc297b009bbe9a5daf083d80137bad1b09e196c6e2f9c1d08250d8b6: Status 404 returned error can't find the container with id 810843c5fc297b009bbe9a5daf083d80137bad1b09e196c6e2f9c1d08250d8b6
Jul 08 16:34:27.011151 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:27.011119    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b9bc9aa_a1bc_4af2_8dff_21213565dd18.slice/crio-edbdb36a8680a1d583abb875bbbb18cafdc5533efb68173bfb4d494d56abb878 WatchSource:0}: Error finding container edbdb36a8680a1d583abb875bbbb18cafdc5533efb68173bfb4d494d56abb878: Status 404 returned error can't find the container with id edbdb36a8680a1d583abb875bbbb18cafdc5533efb68173bfb4d494d56abb878
Jul 08 16:34:27.012951 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:27.012552    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaea71a23_3932_4b4d_8a12_57ef7d976049.slice/crio-7ee6f653122df4a9df5f14408dc6f01719a81acc9ca37e73dbea648318a3fe3f WatchSource:0}: Error finding container 7ee6f653122df4a9df5f14408dc6f01719a81acc9ca37e73dbea648318a3fe3f: Status 404 returned error can't find the container with id 7ee6f653122df4a9df5f14408dc6f01719a81acc9ca37e73dbea648318a3fe3f
Jul 08 16:34:27.013773 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:34:27.013619    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f67aff_be14_4330_87d6_56c5059cd8f8.slice/crio-619b9cb4f801606a7872161761864d849cab661c25e57d8b6ba813e45b0680dc WatchSource:0}: Error finding container 619b9cb4f801606a7872161761864d849cab661c25e57d8b6ba813e45b0680dc: Status 404 returned error can't find the container with id 619b9cb4f801606a7872161761864d849cab661c25e57d8b6ba813e45b0680dc
Jul 08 16:34:27.271900 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:27.271681    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14d465c-2bea-496d-8112-450e25ae7717-metrics-certs\") pod \"network-metrics-daemon-bxjqc\" (UID: \"d14d465c-2bea-496d-8112-450e25ae7717\") " pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:34:27.272014 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:27.271873    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:27.272014 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:27.272013    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d14d465c-2bea-496d-8112-450e25ae7717-metrics-certs podName:d14d465c-2bea-496d-8112-450e25ae7717 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:29.271995371 +0000 UTC m=+5.985356782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d14d465c-2bea-496d-8112-450e25ae7717-metrics-certs") pod "network-metrics-daemon-bxjqc" (UID: "d14d465c-2bea-496d-8112-450e25ae7717") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:27.372282 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:27.372187    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-original-pull-secret\") pod \"global-pull-secret-syncer-sxzrc\" (UID: \"3cf84d3d-0028-4d7f-b178-c4a0d8391cdc\") " pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:27.372282 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:27.372255    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5zv5\" (UniqueName: \"kubernetes.io/projected/dc585660-f572-4e9e-80b3-63e57d32f369-kube-api-access-w5zv5\") pod \"network-check-target-8m52j\" (UID: \"dc585660-f572-4e9e-80b3-63e57d32f369\") " pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:34:27.372527 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:27.372343    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:27.372527 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:27.372422    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:27.372527 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:27.372435    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-original-pull-secret podName:3cf84d3d-0028-4d7f-b178-c4a0d8391cdc nodeName:}" failed. No retries permitted until 2026-07-08 16:34:29.372418031 +0000 UTC m=+6.085779438 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-original-pull-secret") pod "global-pull-secret-syncer-sxzrc" (UID: "3cf84d3d-0028-4d7f-b178-c4a0d8391cdc") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:27.372527 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:27.372441    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:27.372527 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:27.372455    2565 projected.go:196] Error preparing data for projected volume kube-api-access-w5zv5 for pod openshift-network-diagnostics/network-check-target-8m52j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:27.372527 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:27.372525    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc585660-f572-4e9e-80b3-63e57d32f369-kube-api-access-w5zv5 podName:dc585660-f572-4e9e-80b3-63e57d32f369 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:29.372508507 +0000 UTC m=+6.085869930 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-w5zv5" (UniqueName: "kubernetes.io/projected/dc585660-f572-4e9e-80b3-63e57d32f369-kube-api-access-w5zv5") pod "network-check-target-8m52j" (UID: "dc585660-f572-4e9e-80b3-63e57d32f369") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:27.808170 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:27.808069    2565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 16:29:25 +0000 UTC" deadline="2028-01-24 12:23:22.308157979 +0000 UTC"
Jul 08 16:34:27.808170 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:27.808134    2565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13555h48m54.500028099s"
Jul 08 16:34:27.819136 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:27.818315    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:34:27.819136 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:27.818461    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxjqc" podUID="d14d465c-2bea-496d-8112-450e25ae7717"
Jul 08 16:34:27.819136 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:27.818902    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:34:27.819136 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:27.819009    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m52j" podUID="dc585660-f572-4e9e-80b3-63e57d32f369"
Jul 08 16:34:27.865637 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:27.864717    2565 generic.go:356] "Generic (PLEG): container finished" podID="8217407d855ec6a5ce57220657163aed" containerID="3d6adee1e7b45a4fa1e2410284c453498eab08c95085d72d9ed4d31abb32c785" exitCode=0
Jul 08 16:34:27.865637 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:27.864835    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-142.ec2.internal" event={"ID":"8217407d855ec6a5ce57220657163aed","Type":"ContainerDied","Data":"3d6adee1e7b45a4fa1e2410284c453498eab08c95085d72d9ed4d31abb32c785"}
Jul 08 16:34:27.874481 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:27.874442    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-142.ec2.internal" event={"ID":"6c8c7615c18634239b1c0970c2d029c0","Type":"ContainerStarted","Data":"a36df6455845446013eb6b6248e58fa6d0b66216235152d37a31f368bd7eee76"}
Jul 08 16:34:27.886540 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:27.886506    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lscjn" event={"ID":"6f378482-a771-45ff-85cb-3d9b4b887806","Type":"ContainerStarted","Data":"31f87e3e5d38eb7130782b5e1158a0ceeff3d3c5a6930b7bcac30020952fc94b"}
Jul 08 16:34:27.895272 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:27.895238    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w9jwf" event={"ID":"27f67aff-be14-4330-87d6-56c5059cd8f8","Type":"ContainerStarted","Data":"619b9cb4f801606a7872161761864d849cab661c25e57d8b6ba813e45b0680dc"}
Jul 08 16:34:27.918797 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:27.918720    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t8qnb" event={"ID":"7b9bc9aa-a1bc-4af2-8dff-21213565dd18","Type":"ContainerStarted","Data":"edbdb36a8680a1d583abb875bbbb18cafdc5533efb68173bfb4d494d56abb878"}
Jul 08 16:34:27.920694 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:27.920662    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hbbx2" event={"ID":"15283bbe-ca5b-4f0e-a8fb-03f056b9f0b5","Type":"ContainerStarted","Data":"822dfae992eccbc223ad8cc18317a1b70b21fd12ddc46d40a0a88e11502a98df"}
Jul 08 16:34:27.938122 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:27.938086    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hq9qt" event={"ID":"1d51073c-aced-4933-a4f7-0363e8020735","Type":"ContainerStarted","Data":"f9928bdb9db3f2a94c37084ab1ebca4f0a655736e59644d65fa48eeedb690b59"}
Jul 08 16:34:27.962029 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:27.961995    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd" event={"ID":"aea71a23-3932-4b4d-8a12-57ef7d976049","Type":"ContainerStarted","Data":"7ee6f653122df4a9df5f14408dc6f01719a81acc9ca37e73dbea648318a3fe3f"}
Jul 08 16:34:27.983968 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:27.983903    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-97xhl" event={"ID":"8086c91a-48e3-4158-a6ba-004f9046fa09","Type":"ContainerStarted","Data":"810843c5fc297b009bbe9a5daf083d80137bad1b09e196c6e2f9c1d08250d8b6"}
Jul 08 16:34:27.988727 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:27.988086    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pnv7" event={"ID":"190a9759-19d2-4c1d-abf8-c1197ca4e926","Type":"ContainerStarted","Data":"4351a639bdeeed719a22d99fde583fd628e3eda2d402cdbec2d605be84e7827e"}
Jul 08 16:34:27.998194 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:27.998158    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6qgmf" event={"ID":"42d7b641-e70b-4a9d-b7ee-2acab63c9942","Type":"ContainerStarted","Data":"36a4f51ebc79b55cdd68968f72b41580933fd367e0dd0569cecde5b4db9fac9f"}
Jul 08 16:34:28.816717 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:28.816620    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:28.817245 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:28.816776    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxzrc" podUID="3cf84d3d-0028-4d7f-b178-c4a0d8391cdc"
Jul 08 16:34:29.016734 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:29.016697    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-142.ec2.internal" event={"ID":"8217407d855ec6a5ce57220657163aed","Type":"ContainerStarted","Data":"6f85be84f12b50db0d38eac326a9b71d5eec5ee836c6cf708ca6fd6ddaf8bf94"}
Jul 08 16:34:29.039255 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:29.039137    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-142.ec2.internal" podStartSLOduration=4.039111279 podStartE2EDuration="4.039111279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:27.929386793 +0000 UTC m=+4.642748228" watchObservedRunningTime="2026-07-08 16:34:29.039111279 +0000 UTC m=+5.752472707"
Jul 08 16:34:29.289133 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:29.288729    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14d465c-2bea-496d-8112-450e25ae7717-metrics-certs\") pod \"network-metrics-daemon-bxjqc\" (UID: \"d14d465c-2bea-496d-8112-450e25ae7717\") " pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:34:29.289133 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:29.288878    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:29.289133 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:29.288945    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d14d465c-2bea-496d-8112-450e25ae7717-metrics-certs podName:d14d465c-2bea-496d-8112-450e25ae7717 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:33.288925236 +0000 UTC m=+10.002286658 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d14d465c-2bea-496d-8112-450e25ae7717-metrics-certs") pod "network-metrics-daemon-bxjqc" (UID: "d14d465c-2bea-496d-8112-450e25ae7717") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:29.390933 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:29.390075    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-original-pull-secret\") pod \"global-pull-secret-syncer-sxzrc\" (UID: \"3cf84d3d-0028-4d7f-b178-c4a0d8391cdc\") " pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:29.390933 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:29.390148    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5zv5\" (UniqueName: \"kubernetes.io/projected/dc585660-f572-4e9e-80b3-63e57d32f369-kube-api-access-w5zv5\") pod \"network-check-target-8m52j\" (UID: \"dc585660-f572-4e9e-80b3-63e57d32f369\") " pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:34:29.390933 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:29.390321    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:29.390933 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:29.390342    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:29.390933 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:29.390355    2565 projected.go:196] Error preparing data for projected volume kube-api-access-w5zv5 for pod openshift-network-diagnostics/network-check-target-8m52j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:29.390933 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:29.390412    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc585660-f572-4e9e-80b3-63e57d32f369-kube-api-access-w5zv5 podName:dc585660-f572-4e9e-80b3-63e57d32f369 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:33.390393799 +0000 UTC m=+10.103755223 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-w5zv5" (UniqueName: "kubernetes.io/projected/dc585660-f572-4e9e-80b3-63e57d32f369-kube-api-access-w5zv5") pod "network-check-target-8m52j" (UID: "dc585660-f572-4e9e-80b3-63e57d32f369") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:29.390933 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:29.390805    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:29.390933 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:29.390857    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-original-pull-secret podName:3cf84d3d-0028-4d7f-b178-c4a0d8391cdc nodeName:}" failed. No retries permitted until 2026-07-08 16:34:33.390842617 +0000 UTC m=+10.104204039 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-original-pull-secret") pod "global-pull-secret-syncer-sxzrc" (UID: "3cf84d3d-0028-4d7f-b178-c4a0d8391cdc") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:29.817078 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:29.816256    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:34:29.817078 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:29.816400    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxjqc" podUID="d14d465c-2bea-496d-8112-450e25ae7717"
Jul 08 16:34:29.817078 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:29.816932    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:34:29.817078 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:29.817034    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m52j" podUID="dc585660-f572-4e9e-80b3-63e57d32f369"
Jul 08 16:34:30.817062 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:30.816573    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:30.817062 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:30.816721    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxzrc" podUID="3cf84d3d-0028-4d7f-b178-c4a0d8391cdc"
Jul 08 16:34:31.815974 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:31.815899    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:34:31.816122 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:31.816060    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m52j" podUID="dc585660-f572-4e9e-80b3-63e57d32f369"
Jul 08 16:34:31.816122 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:31.816090    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:34:31.816240 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:31.816212    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxjqc" podUID="d14d465c-2bea-496d-8112-450e25ae7717"
Jul 08 16:34:32.815994 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:32.815968    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:32.816419 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:32.816099    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxzrc" podUID="3cf84d3d-0028-4d7f-b178-c4a0d8391cdc"
Jul 08 16:34:33.323794 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:33.323756    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14d465c-2bea-496d-8112-450e25ae7717-metrics-certs\") pod \"network-metrics-daemon-bxjqc\" (UID: \"d14d465c-2bea-496d-8112-450e25ae7717\") " pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:34:33.323970 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:33.323932    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:33.324056 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:33.323995    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d14d465c-2bea-496d-8112-450e25ae7717-metrics-certs podName:d14d465c-2bea-496d-8112-450e25ae7717 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:41.323977089 +0000 UTC m=+18.037338511 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d14d465c-2bea-496d-8112-450e25ae7717-metrics-certs") pod "network-metrics-daemon-bxjqc" (UID: "d14d465c-2bea-496d-8112-450e25ae7717") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:33.425177 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:33.424359    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-original-pull-secret\") pod \"global-pull-secret-syncer-sxzrc\" (UID: \"3cf84d3d-0028-4d7f-b178-c4a0d8391cdc\") " pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:33.425177 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:33.424434    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5zv5\" (UniqueName: \"kubernetes.io/projected/dc585660-f572-4e9e-80b3-63e57d32f369-kube-api-access-w5zv5\") pod \"network-check-target-8m52j\" (UID: \"dc585660-f572-4e9e-80b3-63e57d32f369\") " pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:34:33.425177 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:33.424594    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:33.425177 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:33.424614    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:33.425177 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:33.424627    2565 projected.go:196] Error preparing data for projected volume kube-api-access-w5zv5 for pod openshift-network-diagnostics/network-check-target-8m52j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:33.425177 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:33.424705    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc585660-f572-4e9e-80b3-63e57d32f369-kube-api-access-w5zv5 podName:dc585660-f572-4e9e-80b3-63e57d32f369 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:41.42466895 +0000 UTC m=+18.138030373 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-w5zv5" (UniqueName: "kubernetes.io/projected/dc585660-f572-4e9e-80b3-63e57d32f369-kube-api-access-w5zv5") pod "network-check-target-8m52j" (UID: "dc585660-f572-4e9e-80b3-63e57d32f369") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:33.425177 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:33.425091    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:33.425177 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:33.425140    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-original-pull-secret podName:3cf84d3d-0028-4d7f-b178-c4a0d8391cdc nodeName:}" failed. No retries permitted until 2026-07-08 16:34:41.42512436 +0000 UTC m=+18.138485782 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-original-pull-secret") pod "global-pull-secret-syncer-sxzrc" (UID: "3cf84d3d-0028-4d7f-b178-c4a0d8391cdc") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:33.822410 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:33.822383    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:34:33.822890 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:33.822502    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m52j" podUID="dc585660-f572-4e9e-80b3-63e57d32f369"
Jul 08 16:34:33.822890 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:33.822535    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:34:33.822890 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:33.822618    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxjqc" podUID="d14d465c-2bea-496d-8112-450e25ae7717"
Jul 08 16:34:34.816269 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:34.816226    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:34.816437 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:34.816366    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxzrc" podUID="3cf84d3d-0028-4d7f-b178-c4a0d8391cdc"
Jul 08 16:34:35.815731 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:35.815657    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:34:35.816194 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:35.815665    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:34:35.816194 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:35.815799    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxjqc" podUID="d14d465c-2bea-496d-8112-450e25ae7717"
Jul 08 16:34:35.816194 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:35.815899    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m52j" podUID="dc585660-f572-4e9e-80b3-63e57d32f369"
Jul 08 16:34:36.816490 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:36.816454    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:36.816975 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:36.816586    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxzrc" podUID="3cf84d3d-0028-4d7f-b178-c4a0d8391cdc"
Jul 08 16:34:37.816208 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:37.816131    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:34:37.816360 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:37.816132    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:34:37.816360 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:37.816284    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxjqc" podUID="d14d465c-2bea-496d-8112-450e25ae7717"
Jul 08 16:34:37.816477 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:37.816349    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m52j" podUID="dc585660-f572-4e9e-80b3-63e57d32f369"
Jul 08 16:34:38.815737 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:38.815701    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:38.816181 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:38.815827    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxzrc" podUID="3cf84d3d-0028-4d7f-b178-c4a0d8391cdc"
Jul 08 16:34:39.816661 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:39.816621    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:34:39.817117 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:39.816631    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:34:39.817117 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:39.816777    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m52j" podUID="dc585660-f572-4e9e-80b3-63e57d32f369"
Jul 08 16:34:39.817117 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:39.816820    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxjqc" podUID="d14d465c-2bea-496d-8112-450e25ae7717"
Jul 08 16:34:40.815770 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:40.815728    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:40.816005 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:40.815871    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxzrc" podUID="3cf84d3d-0028-4d7f-b178-c4a0d8391cdc"
Jul 08 16:34:41.382798 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:41.382760    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14d465c-2bea-496d-8112-450e25ae7717-metrics-certs\") pod \"network-metrics-daemon-bxjqc\" (UID: \"d14d465c-2bea-496d-8112-450e25ae7717\") " pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:34:41.383192 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:41.382900    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:41.383192 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:41.382965    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d14d465c-2bea-496d-8112-450e25ae7717-metrics-certs podName:d14d465c-2bea-496d-8112-450e25ae7717 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:57.382949834 +0000 UTC m=+34.096311239 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d14d465c-2bea-496d-8112-450e25ae7717-metrics-certs") pod "network-metrics-daemon-bxjqc" (UID: "d14d465c-2bea-496d-8112-450e25ae7717") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:41.484005 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:41.483972    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-original-pull-secret\") pod \"global-pull-secret-syncer-sxzrc\" (UID: \"3cf84d3d-0028-4d7f-b178-c4a0d8391cdc\") " pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:41.484153 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:41.484022    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5zv5\" (UniqueName: \"kubernetes.io/projected/dc585660-f572-4e9e-80b3-63e57d32f369-kube-api-access-w5zv5\") pod \"network-check-target-8m52j\" (UID: \"dc585660-f572-4e9e-80b3-63e57d32f369\") " pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:34:41.484153 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:41.484112    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:41.484225 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:41.484175    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-original-pull-secret podName:3cf84d3d-0028-4d7f-b178-c4a0d8391cdc nodeName:}" failed. No retries permitted until 2026-07-08 16:34:57.484159984 +0000 UTC m=+34.197521412 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-original-pull-secret") pod "global-pull-secret-syncer-sxzrc" (UID: "3cf84d3d-0028-4d7f-b178-c4a0d8391cdc") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:41.484225 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:41.484119    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:41.484225 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:41.484206    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:41.484225 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:41.484216    2565 projected.go:196] Error preparing data for projected volume kube-api-access-w5zv5 for pod openshift-network-diagnostics/network-check-target-8m52j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:41.484343 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:41.484246    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc585660-f572-4e9e-80b3-63e57d32f369-kube-api-access-w5zv5 podName:dc585660-f572-4e9e-80b3-63e57d32f369 nodeName:}" failed. No retries permitted until 2026-07-08 16:34:57.484238033 +0000 UTC m=+34.197599438 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-w5zv5" (UniqueName: "kubernetes.io/projected/dc585660-f572-4e9e-80b3-63e57d32f369-kube-api-access-w5zv5") pod "network-check-target-8m52j" (UID: "dc585660-f572-4e9e-80b3-63e57d32f369") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:41.816398 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:41.816366    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:34:41.816568 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:41.816534    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxjqc" podUID="d14d465c-2bea-496d-8112-450e25ae7717"
Jul 08 16:34:41.816627 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:41.816581    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:34:41.816714 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:41.816692    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m52j" podUID="dc585660-f572-4e9e-80b3-63e57d32f369"
Jul 08 16:34:42.816535 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:42.816504    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:42.817018 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:42.816628    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxzrc" podUID="3cf84d3d-0028-4d7f-b178-c4a0d8391cdc"
Jul 08 16:34:43.816400 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:43.816332    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:34:43.816593 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:43.816461    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxjqc" podUID="d14d465c-2bea-496d-8112-450e25ae7717"
Jul 08 16:34:43.816593 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:43.816482    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:34:43.816955 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:43.816610    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m52j" podUID="dc585660-f572-4e9e-80b3-63e57d32f369"
Jul 08 16:34:44.815566 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:44.815536    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:44.815753 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:44.815658    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxzrc" podUID="3cf84d3d-0028-4d7f-b178-c4a0d8391cdc"
Jul 08 16:34:45.820589 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:45.818597    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:34:45.820589 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:45.818963    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxjqc" podUID="d14d465c-2bea-496d-8112-450e25ae7717"
Jul 08 16:34:45.820589 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:45.819048    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:34:45.820589 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:45.819117    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m52j" podUID="dc585660-f572-4e9e-80b3-63e57d32f369"
Jul 08 16:34:46.051854 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:46.051623    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd" event={"ID":"aea71a23-3932-4b4d-8a12-57ef7d976049","Type":"ContainerStarted","Data":"b065a2fc6fa54b8cc8edc5bacf772d22cd57fb23c59b6ff2c8ef83b0880c4fc6"}
Jul 08 16:34:46.051997 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:46.051901    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd" event={"ID":"aea71a23-3932-4b4d-8a12-57ef7d976049","Type":"ContainerStarted","Data":"6736dcd89646eaae53513fb0c0a549ce84fb3a6b9c53b9ad31413b8be8030243"}
Jul 08 16:34:46.053181 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:46.053156    2565 generic.go:356] "Generic (PLEG): container finished" podID="8086c91a-48e3-4158-a6ba-004f9046fa09" containerID="b8e187b81f4a36c2a11af8d6d0ccd7e11820ab2c96c087ea4a2f7b50a376f9a6" exitCode=0
Jul 08 16:34:46.053291 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:46.053231    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-97xhl" event={"ID":"8086c91a-48e3-4158-a6ba-004f9046fa09","Type":"ContainerDied","Data":"b8e187b81f4a36c2a11af8d6d0ccd7e11820ab2c96c087ea4a2f7b50a376f9a6"}
Jul 08 16:34:46.055016 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:46.054995    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pnv7" event={"ID":"190a9759-19d2-4c1d-abf8-c1197ca4e926","Type":"ContainerStarted","Data":"7dc91e0f50b7b57bbb53b352e0b6504d820173f271ea4d5eac8a2fc391f2d81e"}
Jul 08 16:34:46.056468 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:46.056443    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6qgmf" event={"ID":"42d7b641-e70b-4a9d-b7ee-2acab63c9942","Type":"ContainerStarted","Data":"94bc289d8d42e5b85bb97fa431e516301f00c7d350ae19cf24b5d9b16329c528"}
Jul 08 16:34:46.057979 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:46.057954    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lscjn" event={"ID":"6f378482-a771-45ff-85cb-3d9b4b887806","Type":"ContainerStarted","Data":"7e266f3b7bd798ed70108773d56dde56dc41218936cdb72f23e0d037003e0d73"}
Jul 08 16:34:46.059466 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:46.059443    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w9jwf" event={"ID":"27f67aff-be14-4330-87d6-56c5059cd8f8","Type":"ContainerStarted","Data":"c68df59a1bce8e01e18d1153eb95c979c3636e70b1c4715501af70d3cfd24791"}
Jul 08 16:34:46.061281 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:46.061260    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-t8qnb" event={"ID":"7b9bc9aa-a1bc-4af2-8dff-21213565dd18","Type":"ContainerStarted","Data":"3e21653a6250615ab2e5ad8fb005b2a0dc1bfcd5a6f2c5e6fe5925f9b0ba4404"}
Jul 08 16:34:46.062944 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:46.062922    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hq9qt" event={"ID":"1d51073c-aced-4933-a4f7-0363e8020735","Type":"ContainerStarted","Data":"1127fa8b654aae928c3d9887117dc6a9261714cd2cd396bfaea3ab775d851ada"}
Jul 08 16:34:46.080939 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:46.080892    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-142.ec2.internal" podStartSLOduration=21.080877675 podStartE2EDuration="21.080877675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:29.040335411 +0000 UTC m=+5.753696840" watchObservedRunningTime="2026-07-08 16:34:46.080877675 +0000 UTC m=+22.794239103"
Jul 08 16:34:46.100594 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:46.100549    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-t8qnb" podStartSLOduration=3.36560094 podStartE2EDuration="22.100530578s" totalImagesPullingTime="18.734929638s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:46.100367693 +0000 UTC m=+22.813729120" watchObservedRunningTime="2026-07-08 16:34:46.100530578 +0000 UTC m=+22.813892006"
Jul 08 16:34:46.117713 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:46.117652    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6qgmf" podStartSLOduration=11.591583618 podStartE2EDuration="22.117638498s" totalImagesPullingTime="10.52605488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:46.117484085 +0000 UTC m=+22.830845536" watchObservedRunningTime="2026-07-08 16:34:46.117638498 +0000 UTC m=+22.830999924"
Jul 08 16:34:46.138269 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:46.135726    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hq9qt" podStartSLOduration=3.475205683 podStartE2EDuration="22.135707453s" totalImagesPullingTime="18.66050177s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:46.134085672 +0000 UTC m=+22.847447118" watchObservedRunningTime="2026-07-08 16:34:46.135707453 +0000 UTC m=+22.849068881"
Jul 08 16:34:46.159906 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:46.159856    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-w9jwf" podStartSLOduration=3.491920587 podStartE2EDuration="22.159836618s" totalImagesPullingTime="18.667916031s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:46.159794945 +0000 UTC m=+22.873156394" watchObservedRunningTime="2026-07-08 16:34:46.159836618 +0000 UTC m=+22.873198047"
Jul 08 16:34:46.178388 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:46.178351    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lscjn" podStartSLOduration=3.516271514 podStartE2EDuration="22.178334871s" totalImagesPullingTime="18.662063357s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:46.178317136 +0000 UTC m=+22.891678596" watchObservedRunningTime="2026-07-08 16:34:46.178334871 +0000 UTC m=+22.891696301"
Jul 08 16:34:46.254377 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:46.254345    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:46.324303 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:46.324259    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:46.426087 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:46.426048    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:46.816728 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:46.816692    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:46.816929 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:46.816832    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxzrc" podUID="3cf84d3d-0028-4d7f-b178-c4a0d8391cdc"
Jul 08 16:34:47.067633 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:47.067483    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hbbx2" event={"ID":"15283bbe-ca5b-4f0e-a8fb-03f056b9f0b5","Type":"ContainerStarted","Data":"e371c63f14ee4d09cb3610169d44b4ec155f60c9de0c6733b6311cfc6286bd7f"}
Jul 08 16:34:47.071463 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:47.071422    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd" event={"ID":"aea71a23-3932-4b4d-8a12-57ef7d976049","Type":"ContainerStarted","Data":"3837d222dfc78428d29f526c232fe2ee821a8d39c7dd303b72c95930292cbceb"}
Jul 08 16:34:47.071647 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:47.071503    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd" event={"ID":"aea71a23-3932-4b4d-8a12-57ef7d976049","Type":"ContainerStarted","Data":"688f22d9a20088ca7415c5892cb57069003ad9a0aef720f1330aa4abe7bad5fc"}
Jul 08 16:34:47.071647 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:47.071556    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd" event={"ID":"aea71a23-3932-4b4d-8a12-57ef7d976049","Type":"ContainerStarted","Data":"1803da9ec5200fb8f51135e100abb17db656cfbefcacc2128d0e424ca9addffe"}
Jul 08 16:34:47.071647 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:47.071604    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd" event={"ID":"aea71a23-3932-4b4d-8a12-57ef7d976049","Type":"ContainerStarted","Data":"8442366d10a3166539ec925e8da284f969eef09bdfe65b4223901a91ff0d1e1f"}
Jul 08 16:34:47.089600 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:47.089543    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hbbx2" podStartSLOduration=4.411443072 podStartE2EDuration="23.089518424s" totalImagesPullingTime="18.678075352s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:47.086816593 +0000 UTC m=+23.800178022" watchObservedRunningTime="2026-07-08 16:34:47.089518424 +0000 UTC m=+23.802879852"
Jul 08 16:34:47.255794 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:47.255760    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:47.364252 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:47.364163    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:47.606302 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:47.606276    2565 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 08 16:34:47.709741 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:47.709627    2565 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T16:34:47.606293816Z","UUID":"4474655e-2785-40d9-abca-7ddb162e1893","Handler":null,"Name":"","Endpoint":""}
Jul 08 16:34:47.712397 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:47.712361    2565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 08 16:34:47.712534 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:47.712410    2565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 08 16:34:47.815959 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:47.815883    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:34:47.816136 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:47.816105    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxjqc" podUID="d14d465c-2bea-496d-8112-450e25ae7717"
Jul 08 16:34:47.816241 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:47.816219    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:34:47.816364 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:47.816342    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m52j" podUID="dc585660-f572-4e9e-80b3-63e57d32f369"
Jul 08 16:34:48.076500 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:48.076468    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pnv7" event={"ID":"190a9759-19d2-4c1d-abf8-c1197ca4e926","Type":"ContainerStarted","Data":"66d8038e9c87a03ee9c88285e09db71af66d9fb8e768ecb79d541ed8b1e34705"}
Jul 08 16:34:48.816032 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:48.815948    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:48.816191 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:48.816086    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxzrc" podUID="3cf84d3d-0028-4d7f-b178-c4a0d8391cdc"
Jul 08 16:34:48.876080 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:48.876033    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:48.983584 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:48.983554    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:49.309798 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:49.309766    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6qgmf"
Jul 08 16:34:49.310559 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:49.310533    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6qgmf"
Jul 08 16:34:49.816168 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:49.815966    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:34:49.816359 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:49.815966    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:34:49.816359 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:49.816263    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxjqc" podUID="d14d465c-2bea-496d-8112-450e25ae7717"
Jul 08 16:34:49.816359 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:49.816342    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m52j" podUID="dc585660-f572-4e9e-80b3-63e57d32f369"
Jul 08 16:34:49.861420 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:49.861376    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:49.964229 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:49.964191    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:50.080854 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:50.080773    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6qgmf"
Jul 08 16:34:50.081268 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:50.081251    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6qgmf"
Jul 08 16:34:50.157014 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:50.156979    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:50.595962 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:50.595927    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:50.748753 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:50.748724    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:50.816155 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:50.816119    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:50.816331 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:50.816229    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxzrc" podUID="3cf84d3d-0028-4d7f-b178-c4a0d8391cdc"
Jul 08 16:34:51.085749 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:51.085642    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd" event={"ID":"aea71a23-3932-4b4d-8a12-57ef7d976049","Type":"ContainerStarted","Data":"1715562e2376b047ba4c8d4de448b4653cda3ba94342681499da9f33d30587e0"}
Jul 08 16:34:51.087413 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:51.087390    2565 generic.go:356] "Generic (PLEG): container finished" podID="8086c91a-48e3-4158-a6ba-004f9046fa09" containerID="8826fbecb7b2cca535cb5ca008ed596246b56524e12e160b2e708eada884242e" exitCode=0
Jul 08 16:34:51.087538 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:51.087441    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-97xhl" event={"ID":"8086c91a-48e3-4158-a6ba-004f9046fa09","Type":"ContainerDied","Data":"8826fbecb7b2cca535cb5ca008ed596246b56524e12e160b2e708eada884242e"}
Jul 08 16:34:51.089581 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:51.089561    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pnv7" event={"ID":"190a9759-19d2-4c1d-abf8-c1197ca4e926","Type":"ContainerStarted","Data":"ffaaf1f10449bc80c2ced874194446626629d80c253fb7d0bac6e129f4842b5c"}
Jul 08 16:34:51.142191 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:51.142149    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-2pnv7" podStartSLOduration=4.100505277 podStartE2EDuration="27.142135036s" totalImagesPullingTime="23.041629759s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:51.141868776 +0000 UTC m=+27.855230224" watchObservedRunningTime="2026-07-08 16:34:51.142135036 +0000 UTC m=+27.855496459"
Jul 08 16:34:51.816396 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:51.816323    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:34:51.816913 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:51.816329    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:34:51.816913 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:51.816461    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxjqc" podUID="d14d465c-2bea-496d-8112-450e25ae7717"
Jul 08 16:34:51.816913 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:51.816496    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m52j" podUID="dc585660-f572-4e9e-80b3-63e57d32f369"
Jul 08 16:34:52.136429 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:52.136353    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:52.238347 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:52.238314    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:52.816662 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:52.816483    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:52.817117 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:52.816790    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxzrc" podUID="3cf84d3d-0028-4d7f-b178-c4a0d8391cdc"
Jul 08 16:34:53.097044 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:53.097009    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd" event={"ID":"aea71a23-3932-4b4d-8a12-57ef7d976049","Type":"ContainerStarted","Data":"1c688bd8f71f1b96ac2db01b5a0261668dc56de52ba96032adfc93fb757df1fe"}
Jul 08 16:34:53.097287 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:53.097263    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:53.099109 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:53.099078    2565 generic.go:356] "Generic (PLEG): container finished" podID="8086c91a-48e3-4158-a6ba-004f9046fa09" containerID="f2d9332c098f7758682e2c511d64c0c1e2952959865b2085af3ba57c7b44f008" exitCode=0
Jul 08 16:34:53.099230 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:53.099150    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-97xhl" event={"ID":"8086c91a-48e3-4158-a6ba-004f9046fa09","Type":"ContainerDied","Data":"f2d9332c098f7758682e2c511d64c0c1e2952959865b2085af3ba57c7b44f008"}
Jul 08 16:34:53.114005 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:53.113981    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:53.135468 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:53.135421    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd" podStartSLOduration=10.388129302 podStartE2EDuration="29.135407462s" totalImagesPullingTime="18.74727816s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:53.135240222 +0000 UTC m=+29.848601649" watchObservedRunningTime="2026-07-08 16:34:53.135407462 +0000 UTC m=+29.848768889"
Jul 08 16:34:53.789556 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:53.789516    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:53.816612 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:53.816578    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:34:53.816797 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:53.816720    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxjqc" podUID="d14d465c-2bea-496d-8112-450e25ae7717"
Jul 08 16:34:53.817253 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:53.816796    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:34:53.817253 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:53.816914    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m52j" podUID="dc585660-f572-4e9e-80b3-63e57d32f369"
Jul 08 16:34:54.102490 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:54.102409    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:54.102642 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:54.102501    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:54.119438 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:54.119408    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:34:54.815628 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:54.815548    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:54.815786 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:54.815658    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxzrc" podUID="3cf84d3d-0028-4d7f-b178-c4a0d8391cdc"
Jul 08 16:34:55.107102 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:55.106855    2565 generic.go:356] "Generic (PLEG): container finished" podID="8086c91a-48e3-4158-a6ba-004f9046fa09" containerID="6e1444c8fde42047cf8204a4f2cc69105e0ba15d616a1fded5868894b76bfa60" exitCode=0
Jul 08 16:34:55.107102 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:55.106936    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-97xhl" event={"ID":"8086c91a-48e3-4158-a6ba-004f9046fa09","Type":"ContainerDied","Data":"6e1444c8fde42047cf8204a4f2cc69105e0ba15d616a1fded5868894b76bfa60"}
Jul 08 16:34:55.147463 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:55.147435    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sxzrc"]
Jul 08 16:34:55.147598 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:55.147557    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:55.147649 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:55.147633    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxzrc" podUID="3cf84d3d-0028-4d7f-b178-c4a0d8391cdc"
Jul 08 16:34:55.154048 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:55.154022    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bxjqc"]
Jul 08 16:34:55.154215 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:55.154199    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:34:55.154318 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:55.154301    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxjqc" podUID="d14d465c-2bea-496d-8112-450e25ae7717"
Jul 08 16:34:55.165382 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:55.165358    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8m52j"]
Jul 08 16:34:55.165484 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:55.165445    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:34:55.165527 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:55.165512    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m52j" podUID="dc585660-f572-4e9e-80b3-63e57d32f369"
Jul 08 16:34:55.670158 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:55.670119    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:55.792429 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:55.792392    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:56.815982 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:56.815942    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:34:56.816641 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:56.815958    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:56.816641 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:56.816064    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:34:56.816641 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:56.816070    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m52j" podUID="dc585660-f572-4e9e-80b3-63e57d32f369"
Jul 08 16:34:56.816641 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:56.816133    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxzrc" podUID="3cf84d3d-0028-4d7f-b178-c4a0d8391cdc"
Jul 08 16:34:56.816641 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:56.816237    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxjqc" podUID="d14d465c-2bea-496d-8112-450e25ae7717"
Jul 08 16:34:57.277500 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:57.277460    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:57.409333 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:57.409299    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14d465c-2bea-496d-8112-450e25ae7717-metrics-certs\") pod \"network-metrics-daemon-bxjqc\" (UID: \"d14d465c-2bea-496d-8112-450e25ae7717\") " pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:34:57.409515 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:57.409446    2565 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:57.409581 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:57.409519    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d14d465c-2bea-496d-8112-450e25ae7717-metrics-certs podName:d14d465c-2bea-496d-8112-450e25ae7717 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:29.409498565 +0000 UTC m=+66.122859982 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d14d465c-2bea-496d-8112-450e25ae7717-metrics-certs") pod "network-metrics-daemon-bxjqc" (UID: "d14d465c-2bea-496d-8112-450e25ae7717") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 08 16:34:57.510588 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:57.510554    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-original-pull-secret\") pod \"global-pull-secret-syncer-sxzrc\" (UID: \"3cf84d3d-0028-4d7f-b178-c4a0d8391cdc\") " pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:57.510763 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:57.510609    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5zv5\" (UniqueName: \"kubernetes.io/projected/dc585660-f572-4e9e-80b3-63e57d32f369-kube-api-access-w5zv5\") pod \"network-check-target-8m52j\" (UID: \"dc585660-f572-4e9e-80b3-63e57d32f369\") " pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:34:57.510763 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:57.510727    2565 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:57.510763 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:57.510736    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 08 16:34:57.510763 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:57.510755    2565 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 08 16:34:57.510763 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:57.510767    2565 projected.go:196] Error preparing data for projected volume kube-api-access-w5zv5 for pod openshift-network-diagnostics/network-check-target-8m52j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:57.510998 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:57.510798    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-original-pull-secret podName:3cf84d3d-0028-4d7f-b178-c4a0d8391cdc nodeName:}" failed. No retries permitted until 2026-07-08 16:35:29.510779359 +0000 UTC m=+66.224140777 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-original-pull-secret") pod "global-pull-secret-syncer-sxzrc" (UID: "3cf84d3d-0028-4d7f-b178-c4a0d8391cdc") : object "kube-system"/"original-pull-secret" not registered
Jul 08 16:34:57.510998 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:57.510815    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/dc585660-f572-4e9e-80b3-63e57d32f369-kube-api-access-w5zv5 podName:dc585660-f572-4e9e-80b3-63e57d32f369 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:29.510808475 +0000 UTC m=+66.224169887 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-w5zv5" (UniqueName: "kubernetes.io/projected/dc585660-f572-4e9e-80b3-63e57d32f369-kube-api-access-w5zv5") pod "network-check-target-8m52j" (UID: "dc585660-f572-4e9e-80b3-63e57d32f369") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 08 16:34:58.645293 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:58.645231    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:58.816221 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:58.816184    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:34:58.816221 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:58.816219    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:34:58.816482 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:58.816183    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:34:58.816482 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:58.816336    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8m52j" podUID="dc585660-f572-4e9e-80b3-63e57d32f369"
Jul 08 16:34:58.816482 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:58.816418    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-sxzrc" podUID="3cf84d3d-0028-4d7f-b178-c4a0d8391cdc"
Jul 08 16:34:58.816967 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:58.816872    2565 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bxjqc" podUID="d14d465c-2bea-496d-8112-450e25ae7717"
Jul 08 16:34:59.120910 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.120878    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-97xhl" event={"ID":"8086c91a-48e3-4158-a6ba-004f9046fa09","Type":"ContainerStarted","Data":"47858f95a36ef1e21b8bc0d17532a014d3955fcbd01479c58e3b86199b8c3ac2"}
Jul 08 16:34:59.598424 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.598345    2565 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-128-142.ec2.internal" event="NodeReady"
Jul 08 16:34:59.598558 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.598476    2565 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 08 16:34:59.666035 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.665999    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-795cc6499b-97mrt"]
Jul 08 16:34:59.667591 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.667576    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:34:59.673738 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.673534    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 08 16:34:59.673738 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.673642    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 08 16:34:59.673976 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.673873    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-r7vmb\""
Jul 08 16:34:59.673976 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.673920    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 08 16:34:59.674411 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.674386    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tf2qd"]
Jul 08 16:34:59.676168 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.676148    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qgtgq"]
Jul 08 16:34:59.676287 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.676272    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tf2qd"
Jul 08 16:34:59.678193 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.678175    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qgtgq"
Jul 08 16:34:59.679695 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.679667    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 08 16:34:59.679807 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.679703    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qx96f\""
Jul 08 16:34:59.679807 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.679757    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 08 16:34:59.681362 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.681343    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 08 16:34:59.681469 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.681454    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 08 16:34:59.681469 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.681461    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 08 16:34:59.681743 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.681729    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tghtj\""
Jul 08 16:34:59.686523 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.686501    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 08 16:34:59.691267 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.691241    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-795cc6499b-97mrt"]
Jul 08 16:34:59.710968 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.710938    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tf2qd"]
Jul 08 16:34:59.711652 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.711626    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qgtgq"]
Jul 08 16:34:59.826057 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.826022    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-bound-sa-token\") pod \"image-registry-795cc6499b-97mrt\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") " pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:34:59.826252 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.826094    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4fnw\" (UniqueName: \"kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-kube-api-access-p4fnw\") pod \"image-registry-795cc6499b-97mrt\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") " pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:34:59.826252 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.826156    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rvpt\" (UniqueName: \"kubernetes.io/projected/d28f25d1-2748-43e3-95a8-eb74a25f8842-kube-api-access-9rvpt\") pod \"ingress-canary-qgtgq\" (UID: \"d28f25d1-2748-43e3-95a8-eb74a25f8842\") " pod="openshift-ingress-canary/ingress-canary-qgtgq"
Jul 08 16:34:59.826252 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.826187    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-trusted-ca\") pod \"image-registry-795cc6499b-97mrt\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") " pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:34:59.826252 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.826215    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/187d83d2-dbc5-46aa-8e7b-f3df420b474c-metrics-tls\") pod \"dns-default-tf2qd\" (UID: \"187d83d2-dbc5-46aa-8e7b-f3df420b474c\") " pod="openshift-dns/dns-default-tf2qd"
Jul 08 16:34:59.826252 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.826249    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjwpw\" (UniqueName: \"kubernetes.io/projected/187d83d2-dbc5-46aa-8e7b-f3df420b474c-kube-api-access-zjwpw\") pod \"dns-default-tf2qd\" (UID: \"187d83d2-dbc5-46aa-8e7b-f3df420b474c\") " pod="openshift-dns/dns-default-tf2qd"
Jul 08 16:34:59.826507 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.826285    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-ca-trust-extracted\") pod \"image-registry-795cc6499b-97mrt\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") " pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:34:59.826507 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.826326    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-installation-pull-secrets\") pod \"image-registry-795cc6499b-97mrt\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") " pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:34:59.826507 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.826408    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:59.826507 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.826454    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-registry-tls\") pod \"image-registry-795cc6499b-97mrt\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") " pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:34:59.826663 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.826544    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/187d83d2-dbc5-46aa-8e7b-f3df420b474c-tmp-dir\") pod \"dns-default-tf2qd\" (UID: \"187d83d2-dbc5-46aa-8e7b-f3df420b474c\") " pod="openshift-dns/dns-default-tf2qd"
Jul 08 16:34:59.826663 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.826571    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/187d83d2-dbc5-46aa-8e7b-f3df420b474c-config-volume\") pod \"dns-default-tf2qd\" (UID: \"187d83d2-dbc5-46aa-8e7b-f3df420b474c\") " pod="openshift-dns/dns-default-tf2qd"
Jul 08 16:34:59.826663 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.826585    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d28f25d1-2748-43e3-95a8-eb74a25f8842-cert\") pod \"ingress-canary-qgtgq\" (UID: \"d28f25d1-2748-43e3-95a8-eb74a25f8842\") " pod="openshift-ingress-canary/ingress-canary-qgtgq"
Jul 08 16:34:59.826663 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.826622    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-registry-certificates\") pod \"image-registry-795cc6499b-97mrt\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") " pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:34:59.826663 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.826653    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-image-registry-private-configuration\") pod \"image-registry-795cc6499b-97mrt\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") " pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:34:59.921966 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.921936    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:34:59.927904 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.927873    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-trusted-ca\") pod \"image-registry-795cc6499b-97mrt\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") " pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:34:59.928009 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.927913    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/187d83d2-dbc5-46aa-8e7b-f3df420b474c-metrics-tls\") pod \"dns-default-tf2qd\" (UID: \"187d83d2-dbc5-46aa-8e7b-f3df420b474c\") " pod="openshift-dns/dns-default-tf2qd"
Jul 08 16:34:59.928009 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.927938    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjwpw\" (UniqueName: \"kubernetes.io/projected/187d83d2-dbc5-46aa-8e7b-f3df420b474c-kube-api-access-zjwpw\") pod \"dns-default-tf2qd\" (UID: \"187d83d2-dbc5-46aa-8e7b-f3df420b474c\") " pod="openshift-dns/dns-default-tf2qd"
Jul 08 16:34:59.928009 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.927960    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-ca-trust-extracted\") pod \"image-registry-795cc6499b-97mrt\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") " pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:34:59.928009 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.927983    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-installation-pull-secrets\") pod \"image-registry-795cc6499b-97mrt\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") " pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:34:59.928211 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.928026    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-registry-tls\") pod \"image-registry-795cc6499b-97mrt\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") " pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:34:59.928211 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:59.928038    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:34:59.928211 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.928090    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/187d83d2-dbc5-46aa-8e7b-f3df420b474c-tmp-dir\") pod \"dns-default-tf2qd\" (UID: \"187d83d2-dbc5-46aa-8e7b-f3df420b474c\") " pod="openshift-dns/dns-default-tf2qd"
Jul 08 16:34:59.928211 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.928119    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/187d83d2-dbc5-46aa-8e7b-f3df420b474c-config-volume\") pod \"dns-default-tf2qd\" (UID: \"187d83d2-dbc5-46aa-8e7b-f3df420b474c\") " pod="openshift-dns/dns-default-tf2qd"
Jul 08 16:34:59.928211 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.928143    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d28f25d1-2748-43e3-95a8-eb74a25f8842-cert\") pod \"ingress-canary-qgtgq\" (UID: \"d28f25d1-2748-43e3-95a8-eb74a25f8842\") " pod="openshift-ingress-canary/ingress-canary-qgtgq"
Jul 08 16:34:59.928211 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.928166    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-registry-certificates\") pod \"image-registry-795cc6499b-97mrt\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") " pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:34:59.928211 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:59.928173    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:34:59.928211 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:59.928193    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-795cc6499b-97mrt: secret "image-registry-tls" not found
Jul 08 16:34:59.928211 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.928196    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-image-registry-private-configuration\") pod \"image-registry-795cc6499b-97mrt\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") " pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:34:59.928595 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.928417    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-ca-trust-extracted\") pod \"image-registry-795cc6499b-97mrt\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") " pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:34:59.928595 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:59.928476    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-registry-tls podName:0483b06a-6fc8-4e43-a1f8-99a63e9e61cb nodeName:}" failed. No retries permitted until 2026-07-08 16:35:00.428453022 +0000 UTC m=+37.141814428 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-registry-tls") pod "image-registry-795cc6499b-97mrt" (UID: "0483b06a-6fc8-4e43-a1f8-99a63e9e61cb") : secret "image-registry-tls" not found
Jul 08 16:34:59.928595 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:34:59.928495    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/187d83d2-dbc5-46aa-8e7b-f3df420b474c-metrics-tls podName:187d83d2-dbc5-46aa-8e7b-f3df420b474c nodeName:}" failed. No retries permitted until 2026-07-08 16:35:00.42848642 +0000 UTC m=+37.141847825 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/187d83d2-dbc5-46aa-8e7b-f3df420b474c-metrics-tls") pod "dns-default-tf2qd" (UID: "187d83d2-dbc5-46aa-8e7b-f3df420b474c") : secret "dns-default-metrics-tls" not found
Jul 08 16:34:59.928595 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.928549    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-bound-sa-token\") pod \"image-registry-795cc6499b-97mrt\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") " pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:34:59.928595 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.928579    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4fnw\" (UniqueName: \"kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-kube-api-access-p4fnw\") pod \"image-registry-795cc6499b-97mrt\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") " pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:34:59.928856 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.928614    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rvpt\" (UniqueName: \"kubernetes.io/projected/d28f25d1-2748-43e3-95a8-eb74a25f8842-kube-api-access-9rvpt\") pod \"ingress-canary-qgtgq\" (UID: \"d28f25d1-2748-43e3-95a8-eb74a25f8842\") " pod="openshift-ingress-canary/ingress-canary-qgtgq"
Jul 08 16:34:59.928856 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.928755    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-registry-certificates\") pod \"image-registry-795cc6499b-97mrt\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") " pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:34:59.928856 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.928797    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/187d83d2-dbc5-46aa-8e7b-f3df420b474c-config-volume\") pod \"dns-default-tf2qd\" (UID: \"187d83d2-dbc5-46aa-8e7b-f3df420b474c\") " pod="openshift-dns/dns-default-tf2qd"
Jul 08 16:34:59.929107 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.929088    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/187d83d2-dbc5-46aa-8e7b-f3df420b474c-tmp-dir\") pod \"dns-default-tf2qd\" (UID: \"187d83d2-dbc5-46aa-8e7b-f3df420b474c\") " pod="openshift-dns/dns-default-tf2qd"
Jul 08 16:34:59.929243 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.929219    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-trusted-ca\") pod \"image-registry-795cc6499b-97mrt\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") " pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:34:59.932351 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.932329    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-image-registry-private-configuration\") pod \"image-registry-795cc6499b-97mrt\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") " pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:34:59.932454 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.932354    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-installation-pull-secrets\") pod \"image-registry-795cc6499b-97mrt\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") " pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:34:59.932454 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.932402    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d28f25d1-2748-43e3-95a8-eb74a25f8842-cert\") pod \"ingress-canary-qgtgq\" (UID: \"d28f25d1-2748-43e3-95a8-eb74a25f8842\") " pod="openshift-ingress-canary/ingress-canary-qgtgq"
Jul 08 16:34:59.944878 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.944852    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjwpw\" (UniqueName: \"kubernetes.io/projected/187d83d2-dbc5-46aa-8e7b-f3df420b474c-kube-api-access-zjwpw\") pod \"dns-default-tf2qd\" (UID: \"187d83d2-dbc5-46aa-8e7b-f3df420b474c\") " pod="openshift-dns/dns-default-tf2qd"
Jul 08 16:34:59.946079 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.946049    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-bound-sa-token\") pod \"image-registry-795cc6499b-97mrt\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") " pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:34:59.946998 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.946974    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rvpt\" (UniqueName: \"kubernetes.io/projected/d28f25d1-2748-43e3-95a8-eb74a25f8842-kube-api-access-9rvpt\") pod \"ingress-canary-qgtgq\" (UID: \"d28f25d1-2748-43e3-95a8-eb74a25f8842\") " pod="openshift-ingress-canary/ingress-canary-qgtgq"
Jul 08 16:34:59.947072 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.947046    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4fnw\" (UniqueName: \"kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-kube-api-access-p4fnw\") pod \"image-registry-795cc6499b-97mrt\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") " pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:34:59.989869 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:34:59.989828    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qgtgq"
Jul 08 16:35:00.126350 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:00.126149    2565 generic.go:356] "Generic (PLEG): container finished" podID="8086c91a-48e3-4158-a6ba-004f9046fa09" containerID="47858f95a36ef1e21b8bc0d17532a014d3955fcbd01479c58e3b86199b8c3ac2" exitCode=0
Jul 08 16:35:00.126495 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:00.126241    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-97xhl" event={"ID":"8086c91a-48e3-4158-a6ba-004f9046fa09","Type":"ContainerDied","Data":"47858f95a36ef1e21b8bc0d17532a014d3955fcbd01479c58e3b86199b8c3ac2"}
Jul 08 16:35:00.126495 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:00.126403    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qgtgq"]
Jul 08 16:35:00.131854 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:35:00.131830    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28f25d1_2748_43e3_95a8_eb74a25f8842.slice/crio-3ba6d10ed2e198eb830e94ba02983e28b234ce7edeb658d1125058478e541522 WatchSource:0}: Error finding container 3ba6d10ed2e198eb830e94ba02983e28b234ce7edeb658d1125058478e541522: Status 404 returned error can't find the container with id 3ba6d10ed2e198eb830e94ba02983e28b234ce7edeb658d1125058478e541522
Jul 08 16:35:00.432934 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:00.432846    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/187d83d2-dbc5-46aa-8e7b-f3df420b474c-metrics-tls\") pod \"dns-default-tf2qd\" (UID: \"187d83d2-dbc5-46aa-8e7b-f3df420b474c\") " pod="openshift-dns/dns-default-tf2qd"
Jul 08 16:35:00.432934 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:00.432887    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-registry-tls\") pod \"image-registry-795cc6499b-97mrt\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") " pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:35:00.433141 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:00.432991    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:35:00.433141 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:00.433002    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-795cc6499b-97mrt: secret "image-registry-tls" not found
Jul 08 16:35:00.433141 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:00.433004    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:35:00.433141 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:00.433057    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-registry-tls podName:0483b06a-6fc8-4e43-a1f8-99a63e9e61cb nodeName:}" failed. No retries permitted until 2026-07-08 16:35:01.433040581 +0000 UTC m=+38.146402003 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-registry-tls") pod "image-registry-795cc6499b-97mrt" (UID: "0483b06a-6fc8-4e43-a1f8-99a63e9e61cb") : secret "image-registry-tls" not found
Jul 08 16:35:00.433141 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:00.433071    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/187d83d2-dbc5-46aa-8e7b-f3df420b474c-metrics-tls podName:187d83d2-dbc5-46aa-8e7b-f3df420b474c nodeName:}" failed. No retries permitted until 2026-07-08 16:35:01.433065019 +0000 UTC m=+38.146426424 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/187d83d2-dbc5-46aa-8e7b-f3df420b474c-metrics-tls") pod "dns-default-tf2qd" (UID: "187d83d2-dbc5-46aa-8e7b-f3df420b474c") : secret "dns-default-metrics-tls" not found
Jul 08 16:35:00.815951 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:00.815861    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:35:00.815951 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:00.815888    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:35:00.816616 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:00.815875    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:35:00.819067 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:00.819032    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 08 16:35:00.819212 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:00.819087    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g78k4\""
Jul 08 16:35:00.819282 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:00.819213    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 08 16:35:00.819282 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:00.819240    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 08 16:35:00.819282 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:00.819252    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zs6bx\""
Jul 08 16:35:00.819907 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:00.819888    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 08 16:35:01.133003 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:01.132922    2565 generic.go:356] "Generic (PLEG): container finished" podID="8086c91a-48e3-4158-a6ba-004f9046fa09" containerID="fc8763e449d18136a4deb85ecc39354f490b64e18bfefccb866837f9c25f0baa" exitCode=0
Jul 08 16:35:01.133166 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:01.133006    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-97xhl" event={"ID":"8086c91a-48e3-4158-a6ba-004f9046fa09","Type":"ContainerDied","Data":"fc8763e449d18136a4deb85ecc39354f490b64e18bfefccb866837f9c25f0baa"}
Jul 08 16:35:01.134299 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:01.134270    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qgtgq" event={"ID":"d28f25d1-2748-43e3-95a8-eb74a25f8842","Type":"ContainerStarted","Data":"3ba6d10ed2e198eb830e94ba02983e28b234ce7edeb658d1125058478e541522"}
Jul 08 16:35:01.441829 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:01.441797    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/187d83d2-dbc5-46aa-8e7b-f3df420b474c-metrics-tls\") pod \"dns-default-tf2qd\" (UID: \"187d83d2-dbc5-46aa-8e7b-f3df420b474c\") " pod="openshift-dns/dns-default-tf2qd"
Jul 08 16:35:01.441987 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:01.441846    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-registry-tls\") pod \"image-registry-795cc6499b-97mrt\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") " pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:35:01.441987 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:01.441928    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:35:01.441987 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:01.441955    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:35:01.441987 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:01.441969    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-795cc6499b-97mrt: secret "image-registry-tls" not found
Jul 08 16:35:01.442211 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:01.442027    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/187d83d2-dbc5-46aa-8e7b-f3df420b474c-metrics-tls podName:187d83d2-dbc5-46aa-8e7b-f3df420b474c nodeName:}" failed. No retries permitted until 2026-07-08 16:35:03.44200236 +0000 UTC m=+40.155363770 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/187d83d2-dbc5-46aa-8e7b-f3df420b474c-metrics-tls") pod "dns-default-tf2qd" (UID: "187d83d2-dbc5-46aa-8e7b-f3df420b474c") : secret "dns-default-metrics-tls" not found
Jul 08 16:35:01.442211 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:01.442049    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-registry-tls podName:0483b06a-6fc8-4e43-a1f8-99a63e9e61cb nodeName:}" failed. No retries permitted until 2026-07-08 16:35:03.442039751 +0000 UTC m=+40.155401171 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-registry-tls") pod "image-registry-795cc6499b-97mrt" (UID: "0483b06a-6fc8-4e43-a1f8-99a63e9e61cb") : secret "image-registry-tls" not found
Jul 08 16:35:01.489839 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:01.489795    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:01.657427 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:01.657391    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:02.139387 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:02.139357    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-97xhl" event={"ID":"8086c91a-48e3-4158-a6ba-004f9046fa09","Type":"ContainerStarted","Data":"e7f37e5a1c816dc03017f3892bd431ba276d2580360847202e87fb70e303b353"}
Jul 08 16:35:02.167536 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:02.167493    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-97xhl" podStartSLOduration=8.644766688 podStartE2EDuration="38.167479365s" totalImagesPullingTime="29.522712677s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:02.167245521 +0000 UTC m=+38.880606968" watchObservedRunningTime="2026-07-08 16:35:02.167479365 +0000 UTC m=+38.880840793"
Jul 08 16:35:02.774371 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:02.774096    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:02.840690 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:02.840641    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:03.030798 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:03.030721    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:03.131616 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:03.131589    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:03.143423 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:03.143400    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qgtgq" event={"ID":"d28f25d1-2748-43e3-95a8-eb74a25f8842","Type":"ContainerStarted","Data":"90553cd8727bb2f411be8e01d9b97347f30c7b2d6c44e08e5d6a4c81bce45191"}
Jul 08 16:35:03.165006 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:03.164946    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qgtgq" podStartSLOduration=1.8006394989999999 podStartE2EDuration="4.16493318s" totalImagesPullingTime="2.364293681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:03.163763328 +0000 UTC m=+39.877124768" watchObservedRunningTime="2026-07-08 16:35:03.16493318 +0000 UTC m=+39.878294606"
Jul 08 16:35:03.458879 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:03.458843    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/187d83d2-dbc5-46aa-8e7b-f3df420b474c-metrics-tls\") pod \"dns-default-tf2qd\" (UID: \"187d83d2-dbc5-46aa-8e7b-f3df420b474c\") " pod="openshift-dns/dns-default-tf2qd"
Jul 08 16:35:03.459030 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:03.458885    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-registry-tls\") pod \"image-registry-795cc6499b-97mrt\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") " pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:35:03.459030 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:03.458988    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:35:03.459030 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:03.459003    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:35:03.459030 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:03.459013    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-795cc6499b-97mrt: secret "image-registry-tls" not found
Jul 08 16:35:03.459153 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:03.459049    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/187d83d2-dbc5-46aa-8e7b-f3df420b474c-metrics-tls podName:187d83d2-dbc5-46aa-8e7b-f3df420b474c nodeName:}" failed. No retries permitted until 2026-07-08 16:35:07.459032905 +0000 UTC m=+44.172394309 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/187d83d2-dbc5-46aa-8e7b-f3df420b474c-metrics-tls") pod "dns-default-tf2qd" (UID: "187d83d2-dbc5-46aa-8e7b-f3df420b474c") : secret "dns-default-metrics-tls" not found
Jul 08 16:35:03.459153 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:03.459064    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-registry-tls podName:0483b06a-6fc8-4e43-a1f8-99a63e9e61cb nodeName:}" failed. No retries permitted until 2026-07-08 16:35:07.459057957 +0000 UTC m=+44.172419362 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-registry-tls") pod "image-registry-795cc6499b-97mrt" (UID: "0483b06a-6fc8-4e43-a1f8-99a63e9e61cb") : secret "image-registry-tls" not found
Jul 08 16:35:03.485975 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:03.485938    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-24ccq"]
Jul 08 16:35:03.507381 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:03.507351    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-24ccq"]
Jul 08 16:35:03.507541 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:03.507440    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-24ccq"
Jul 08 16:35:03.510771 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:03.510732    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 08 16:35:03.511187 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:03.511165    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 08 16:35:03.511715 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:03.511587    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 08 16:35:03.511715 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:03.511633    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 08 16:35:03.511910 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:03.511757    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:03.512568 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:03.512547    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cshmv\""
Jul 08 16:35:03.660117 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:03.660079    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59fa9d8c-d7a1-430d-bd58-560305b06ec9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24ccq\" (UID: \"59fa9d8c-d7a1-430d-bd58-560305b06ec9\") " pod="openshift-insights/insights-runtime-extractor-24ccq"
Jul 08 16:35:03.660297 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:03.660122    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59fa9d8c-d7a1-430d-bd58-560305b06ec9-data-volume\") pod \"insights-runtime-extractor-24ccq\" (UID: \"59fa9d8c-d7a1-430d-bd58-560305b06ec9\") " pod="openshift-insights/insights-runtime-extractor-24ccq"
Jul 08 16:35:03.660297 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:03.660164    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wszq4\" (UniqueName: \"kubernetes.io/projected/59fa9d8c-d7a1-430d-bd58-560305b06ec9-kube-api-access-wszq4\") pod \"insights-runtime-extractor-24ccq\" (UID: \"59fa9d8c-d7a1-430d-bd58-560305b06ec9\") " pod="openshift-insights/insights-runtime-extractor-24ccq"
Jul 08 16:35:03.660297 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:03.660233    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59fa9d8c-d7a1-430d-bd58-560305b06ec9-crio-socket\") pod \"insights-runtime-extractor-24ccq\" (UID: \"59fa9d8c-d7a1-430d-bd58-560305b06ec9\") " pod="openshift-insights/insights-runtime-extractor-24ccq"
Jul 08 16:35:03.660297 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:03.660272    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59fa9d8c-d7a1-430d-bd58-560305b06ec9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24ccq\" (UID: \"59fa9d8c-d7a1-430d-bd58-560305b06ec9\") " pod="openshift-insights/insights-runtime-extractor-24ccq"
Jul 08 16:35:03.761186 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:03.761107    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59fa9d8c-d7a1-430d-bd58-560305b06ec9-crio-socket\") pod \"insights-runtime-extractor-24ccq\" (UID: \"59fa9d8c-d7a1-430d-bd58-560305b06ec9\") " pod="openshift-insights/insights-runtime-extractor-24ccq"
Jul 08 16:35:03.761186 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:03.761165    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59fa9d8c-d7a1-430d-bd58-560305b06ec9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24ccq\" (UID: \"59fa9d8c-d7a1-430d-bd58-560305b06ec9\") " pod="openshift-insights/insights-runtime-extractor-24ccq"
Jul 08 16:35:03.761392 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:03.761227    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59fa9d8c-d7a1-430d-bd58-560305b06ec9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24ccq\" (UID: \"59fa9d8c-d7a1-430d-bd58-560305b06ec9\") " pod="openshift-insights/insights-runtime-extractor-24ccq"
Jul 08 16:35:03.761392 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:03.761236    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 16:35:03.761392 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:03.761259    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59fa9d8c-d7a1-430d-bd58-560305b06ec9-data-volume\") pod \"insights-runtime-extractor-24ccq\" (UID: \"59fa9d8c-d7a1-430d-bd58-560305b06ec9\") " pod="openshift-insights/insights-runtime-extractor-24ccq"
Jul 08 16:35:03.761392 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:03.761281    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wszq4\" (UniqueName: \"kubernetes.io/projected/59fa9d8c-d7a1-430d-bd58-560305b06ec9-kube-api-access-wszq4\") pod \"insights-runtime-extractor-24ccq\" (UID: \"59fa9d8c-d7a1-430d-bd58-560305b06ec9\") " pod="openshift-insights/insights-runtime-extractor-24ccq"
Jul 08 16:35:03.761392 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:03.761300    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/59fa9d8c-d7a1-430d-bd58-560305b06ec9-kube-rbac-proxy-cm podName:59fa9d8c-d7a1-430d-bd58-560305b06ec9 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:04.261279968 +0000 UTC m=+40.974641463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/59fa9d8c-d7a1-430d-bd58-560305b06ec9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-24ccq" (UID: "59fa9d8c-d7a1-430d-bd58-560305b06ec9") : configmap "kube-rbac-proxy" not found
Jul 08 16:35:03.761392 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:03.761297    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/59fa9d8c-d7a1-430d-bd58-560305b06ec9-crio-socket\") pod \"insights-runtime-extractor-24ccq\" (UID: \"59fa9d8c-d7a1-430d-bd58-560305b06ec9\") " pod="openshift-insights/insights-runtime-extractor-24ccq"
Jul 08 16:35:03.761392 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:03.761373    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:03.761648 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:03.761410    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59fa9d8c-d7a1-430d-bd58-560305b06ec9-insights-runtime-extractor-tls podName:59fa9d8c-d7a1-430d-bd58-560305b06ec9 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:04.261398859 +0000 UTC m=+40.974760266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/59fa9d8c-d7a1-430d-bd58-560305b06ec9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-24ccq" (UID: "59fa9d8c-d7a1-430d-bd58-560305b06ec9") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:03.761648 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:03.761557    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/59fa9d8c-d7a1-430d-bd58-560305b06ec9-data-volume\") pod \"insights-runtime-extractor-24ccq\" (UID: \"59fa9d8c-d7a1-430d-bd58-560305b06ec9\") " pod="openshift-insights/insights-runtime-extractor-24ccq"
Jul 08 16:35:03.772031 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:03.772009    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wszq4\" (UniqueName: \"kubernetes.io/projected/59fa9d8c-d7a1-430d-bd58-560305b06ec9-kube-api-access-wszq4\") pod \"insights-runtime-extractor-24ccq\" (UID: \"59fa9d8c-d7a1-430d-bd58-560305b06ec9\") " pod="openshift-insights/insights-runtime-extractor-24ccq"
Jul 08 16:35:03.979110 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:03.979077    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:04.152980 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:04.152884    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:04.251908 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:04.251866    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:04.265916 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:04.265895    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59fa9d8c-d7a1-430d-bd58-560305b06ec9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24ccq\" (UID: \"59fa9d8c-d7a1-430d-bd58-560305b06ec9\") " pod="openshift-insights/insights-runtime-extractor-24ccq"
Jul 08 16:35:04.266040 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:04.266023    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 16:35:04.266120 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:04.266098    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59fa9d8c-d7a1-430d-bd58-560305b06ec9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24ccq\" (UID: \"59fa9d8c-d7a1-430d-bd58-560305b06ec9\") " pod="openshift-insights/insights-runtime-extractor-24ccq"
Jul 08 16:35:04.266190 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:04.266152    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/59fa9d8c-d7a1-430d-bd58-560305b06ec9-kube-rbac-proxy-cm podName:59fa9d8c-d7a1-430d-bd58-560305b06ec9 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:05.266136717 +0000 UTC m=+41.979498123 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/59fa9d8c-d7a1-430d-bd58-560305b06ec9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-24ccq" (UID: "59fa9d8c-d7a1-430d-bd58-560305b06ec9") : configmap "kube-rbac-proxy" not found
Jul 08 16:35:04.266262 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:04.266246    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:04.266319 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:04.266309    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59fa9d8c-d7a1-430d-bd58-560305b06ec9-insights-runtime-extractor-tls podName:59fa9d8c-d7a1-430d-bd58-560305b06ec9 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:05.266294957 +0000 UTC m=+41.979656378 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/59fa9d8c-d7a1-430d-bd58-560305b06ec9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-24ccq" (UID: "59fa9d8c-d7a1-430d-bd58-560305b06ec9") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:04.414084 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:04.413980    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:04.482159 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:04.482130    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:04.676829 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:04.676799    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:04.783802 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:04.783771    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:05.274206 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:05.274168    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59fa9d8c-d7a1-430d-bd58-560305b06ec9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24ccq\" (UID: \"59fa9d8c-d7a1-430d-bd58-560305b06ec9\") " pod="openshift-insights/insights-runtime-extractor-24ccq"
Jul 08 16:35:05.274647 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:05.274294    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59fa9d8c-d7a1-430d-bd58-560305b06ec9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24ccq\" (UID: \"59fa9d8c-d7a1-430d-bd58-560305b06ec9\") " pod="openshift-insights/insights-runtime-extractor-24ccq"
Jul 08 16:35:05.274647 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:05.274327    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:05.274647 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:05.274391    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59fa9d8c-d7a1-430d-bd58-560305b06ec9-insights-runtime-extractor-tls podName:59fa9d8c-d7a1-430d-bd58-560305b06ec9 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:07.274373063 +0000 UTC m=+43.987734469 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/59fa9d8c-d7a1-430d-bd58-560305b06ec9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-24ccq" (UID: "59fa9d8c-d7a1-430d-bd58-560305b06ec9") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:05.274647 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:05.274404    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 16:35:05.274647 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:05.274458    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/59fa9d8c-d7a1-430d-bd58-560305b06ec9-kube-rbac-proxy-cm podName:59fa9d8c-d7a1-430d-bd58-560305b06ec9 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:07.274442227 +0000 UTC m=+43.987803647 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/59fa9d8c-d7a1-430d-bd58-560305b06ec9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-24ccq" (UID: "59fa9d8c-d7a1-430d-bd58-560305b06ec9") : configmap "kube-rbac-proxy" not found
Jul 08 16:35:05.458999 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:05.458971    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hq9qt_1d51073c-aced-4933-a4f7-0363e8020735/dns-node-resolver/0.log"
Jul 08 16:35:05.773073 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:05.773038    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:06.262420 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:06.262392    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lscjn_6f378482-a771-45ff-85cb-3d9b4b887806/node-ca/0.log"
Jul 08 16:35:06.262420 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:06.262421    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-f4js8"]
Jul 08 16:35:06.293037 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:06.293011    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-f4js8"]
Jul 08 16:35:06.293434 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:06.293117    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-f4js8"
Jul 08 16:35:06.296154 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:06.296134    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 08 16:35:06.296355 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:06.296335    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 08 16:35:06.296422 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:06.296339    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 08 16:35:06.297271 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:06.297248    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-grsgs\""
Jul 08 16:35:06.297354 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:06.297254    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 08 16:35:06.339458 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:06.339427    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:06.382488 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:06.382458    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/328d898b-11f5-49c2-9bbb-933db5c92151-signing-key\") pod \"service-ca-758f599c5-f4js8\" (UID: \"328d898b-11f5-49c2-9bbb-933db5c92151\") " pod="openshift-service-ca/service-ca-758f599c5-f4js8"
Jul 08 16:35:06.382591 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:06.382498    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/328d898b-11f5-49c2-9bbb-933db5c92151-signing-cabundle\") pod \"service-ca-758f599c5-f4js8\" (UID: \"328d898b-11f5-49c2-9bbb-933db5c92151\") " pod="openshift-service-ca/service-ca-758f599c5-f4js8"
Jul 08 16:35:06.382591 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:06.382525    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-977fp\" (UniqueName: \"kubernetes.io/projected/328d898b-11f5-49c2-9bbb-933db5c92151-kube-api-access-977fp\") pod \"service-ca-758f599c5-f4js8\" (UID: \"328d898b-11f5-49c2-9bbb-933db5c92151\") " pod="openshift-service-ca/service-ca-758f599c5-f4js8"
Jul 08 16:35:06.415907 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:06.415864    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:06.482979 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:06.482941    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/328d898b-11f5-49c2-9bbb-933db5c92151-signing-key\") pod \"service-ca-758f599c5-f4js8\" (UID: \"328d898b-11f5-49c2-9bbb-933db5c92151\") " pod="openshift-service-ca/service-ca-758f599c5-f4js8"
Jul 08 16:35:06.482979 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:06.482985    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/328d898b-11f5-49c2-9bbb-933db5c92151-signing-cabundle\") pod \"service-ca-758f599c5-f4js8\" (UID: \"328d898b-11f5-49c2-9bbb-933db5c92151\") " pod="openshift-service-ca/service-ca-758f599c5-f4js8"
Jul 08 16:35:06.483171 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:06.483014    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-977fp\" (UniqueName: \"kubernetes.io/projected/328d898b-11f5-49c2-9bbb-933db5c92151-kube-api-access-977fp\") pod \"service-ca-758f599c5-f4js8\" (UID: \"328d898b-11f5-49c2-9bbb-933db5c92151\") " pod="openshift-service-ca/service-ca-758f599c5-f4js8"
Jul 08 16:35:06.483633 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:06.483609    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/328d898b-11f5-49c2-9bbb-933db5c92151-signing-cabundle\") pod \"service-ca-758f599c5-f4js8\" (UID: \"328d898b-11f5-49c2-9bbb-933db5c92151\") " pod="openshift-service-ca/service-ca-758f599c5-f4js8"
Jul 08 16:35:06.485354 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:06.485327    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/328d898b-11f5-49c2-9bbb-933db5c92151-signing-key\") pod \"service-ca-758f599c5-f4js8\" (UID: \"328d898b-11f5-49c2-9bbb-933db5c92151\") " pod="openshift-service-ca/service-ca-758f599c5-f4js8"
Jul 08 16:35:06.491931 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:06.491910    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-977fp\" (UniqueName: \"kubernetes.io/projected/328d898b-11f5-49c2-9bbb-933db5c92151-kube-api-access-977fp\") pod \"service-ca-758f599c5-f4js8\" (UID: \"328d898b-11f5-49c2-9bbb-933db5c92151\") " pod="openshift-service-ca/service-ca-758f599c5-f4js8"
Jul 08 16:35:06.602633 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:06.602549    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-f4js8"
Jul 08 16:35:06.748836 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:06.748807    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-f4js8"]
Jul 08 16:35:06.751753 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:35:06.751720    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod328d898b_11f5_49c2_9bbb_933db5c92151.slice/crio-5cb43e518b25b6cb3bf6fa3f3647d53dfed36c7034dd5460df55e96c8b071804 WatchSource:0}: Error finding container 5cb43e518b25b6cb3bf6fa3f3647d53dfed36c7034dd5460df55e96c8b071804: Status 404 returned error can't find the container with id 5cb43e518b25b6cb3bf6fa3f3647d53dfed36c7034dd5460df55e96c8b071804
Jul 08 16:35:07.153591 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:07.153550    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-f4js8" event={"ID":"328d898b-11f5-49c2-9bbb-933db5c92151","Type":"ContainerStarted","Data":"5cb43e518b25b6cb3bf6fa3f3647d53dfed36c7034dd5460df55e96c8b071804"}
Jul 08 16:35:07.288541 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:07.288500    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59fa9d8c-d7a1-430d-bd58-560305b06ec9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24ccq\" (UID: \"59fa9d8c-d7a1-430d-bd58-560305b06ec9\") " pod="openshift-insights/insights-runtime-extractor-24ccq"
Jul 08 16:35:07.288741 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:07.288630    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59fa9d8c-d7a1-430d-bd58-560305b06ec9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24ccq\" (UID: \"59fa9d8c-d7a1-430d-bd58-560305b06ec9\") " pod="openshift-insights/insights-runtime-extractor-24ccq"
Jul 08 16:35:07.288741 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:07.288642    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:07.288741 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:07.288720    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 16:35:07.288741 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:07.288740    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59fa9d8c-d7a1-430d-bd58-560305b06ec9-insights-runtime-extractor-tls podName:59fa9d8c-d7a1-430d-bd58-560305b06ec9 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:11.288717758 +0000 UTC m=+48.002079188 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/59fa9d8c-d7a1-430d-bd58-560305b06ec9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-24ccq" (UID: "59fa9d8c-d7a1-430d-bd58-560305b06ec9") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:07.288933 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:07.288758    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/59fa9d8c-d7a1-430d-bd58-560305b06ec9-kube-rbac-proxy-cm podName:59fa9d8c-d7a1-430d-bd58-560305b06ec9 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:11.28874695 +0000 UTC m=+48.002108375 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/59fa9d8c-d7a1-430d-bd58-560305b06ec9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-24ccq" (UID: "59fa9d8c-d7a1-430d-bd58-560305b06ec9") : configmap "kube-rbac-proxy" not found
Jul 08 16:35:07.422394 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:07.422359    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:07.459374 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:07.459346    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qgtgq_d28f25d1-2748-43e3-95a8-eb74a25f8842/serve-healthcheck-canary/0.log"
Jul 08 16:35:07.490845 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:07.490812    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/187d83d2-dbc5-46aa-8e7b-f3df420b474c-metrics-tls\") pod \"dns-default-tf2qd\" (UID: \"187d83d2-dbc5-46aa-8e7b-f3df420b474c\") " pod="openshift-dns/dns-default-tf2qd"
Jul 08 16:35:07.491040 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:07.490877    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-registry-tls\") pod \"image-registry-795cc6499b-97mrt\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") " pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:35:07.491040 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:07.490981    2565 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 08 16:35:07.491147 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:07.491062    2565 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 08 16:35:07.491147 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:07.491080    2565 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-795cc6499b-97mrt: secret "image-registry-tls" not found
Jul 08 16:35:07.491147 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:07.491064    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/187d83d2-dbc5-46aa-8e7b-f3df420b474c-metrics-tls podName:187d83d2-dbc5-46aa-8e7b-f3df420b474c nodeName:}" failed. No retries permitted until 2026-07-08 16:35:15.49104385 +0000 UTC m=+52.204405266 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/187d83d2-dbc5-46aa-8e7b-f3df420b474c-metrics-tls") pod "dns-default-tf2qd" (UID: "187d83d2-dbc5-46aa-8e7b-f3df420b474c") : secret "dns-default-metrics-tls" not found
Jul 08 16:35:07.491147 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:07.491143    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-registry-tls podName:0483b06a-6fc8-4e43-a1f8-99a63e9e61cb nodeName:}" failed. No retries permitted until 2026-07-08 16:35:15.491127192 +0000 UTC m=+52.204488603 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-registry-tls") pod "image-registry-795cc6499b-97mrt" (UID: "0483b06a-6fc8-4e43-a1f8-99a63e9e61cb") : secret "image-registry-tls" not found
Jul 08 16:35:09.822775 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:09.822730    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:10.060938 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:10.060900    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:10.162360 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:10.162272    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:10.163738 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:10.163703    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-f4js8" event={"ID":"328d898b-11f5-49c2-9bbb-933db5c92151","Type":"ContainerStarted","Data":"a58bdfc0bcfd4967315f28782f91aa087042790ee8cfcbbdc4fa2920c4a07e0d"}
Jul 08 16:35:10.183251 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:10.183189    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-f4js8" podStartSLOduration=1.801071625 podStartE2EDuration="4.183167971s" totalImagesPullingTime="2.382096346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:10.182209 +0000 UTC m=+46.895570444" watchObservedRunningTime="2026-07-08 16:35:10.183167971 +0000 UTC m=+46.896529402"
Jul 08 16:35:10.569759 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:10.569710    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:10.639108 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:10.639077    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:11.322509 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:11.322479    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59fa9d8c-d7a1-430d-bd58-560305b06ec9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24ccq\" (UID: \"59fa9d8c-d7a1-430d-bd58-560305b06ec9\") " pod="openshift-insights/insights-runtime-extractor-24ccq"
Jul 08 16:35:11.322907 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:11.322539    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59fa9d8c-d7a1-430d-bd58-560305b06ec9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24ccq\" (UID: \"59fa9d8c-d7a1-430d-bd58-560305b06ec9\") " pod="openshift-insights/insights-runtime-extractor-24ccq"
Jul 08 16:35:11.322907 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:11.322604    2565 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 08 16:35:11.322907 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:11.322647    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:11.322907 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:11.322669    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/59fa9d8c-d7a1-430d-bd58-560305b06ec9-kube-rbac-proxy-cm podName:59fa9d8c-d7a1-430d-bd58-560305b06ec9 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:19.322653354 +0000 UTC m=+56.036014759 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/59fa9d8c-d7a1-430d-bd58-560305b06ec9-kube-rbac-proxy-cm") pod "insights-runtime-extractor-24ccq" (UID: "59fa9d8c-d7a1-430d-bd58-560305b06ec9") : configmap "kube-rbac-proxy" not found
Jul 08 16:35:11.322907 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:11.322709    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59fa9d8c-d7a1-430d-bd58-560305b06ec9-insights-runtime-extractor-tls podName:59fa9d8c-d7a1-430d-bd58-560305b06ec9 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:19.32269624 +0000 UTC m=+56.036057661 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/59fa9d8c-d7a1-430d-bd58-560305b06ec9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-24ccq" (UID: "59fa9d8c-d7a1-430d-bd58-560305b06ec9") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:11.394901 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:11.394873    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:11.660568 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:11.660496    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:11.932219 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:11.932183    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:12.126451 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:12.126419    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:12.316929 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:12.316839    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:12.720726 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:12.720670    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:13.271690 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:13.271636    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:13.557210 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:13.557122    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:13.657736 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:13.657704    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:13.925319 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:13.925286    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:14.130758 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:14.130727    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:14.196201 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:14.196104    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:14.324504 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:14.324465    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:14.528718 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:14.528613    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:15.560348 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:15.560310    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/187d83d2-dbc5-46aa-8e7b-f3df420b474c-metrics-tls\") pod \"dns-default-tf2qd\" (UID: \"187d83d2-dbc5-46aa-8e7b-f3df420b474c\") " pod="openshift-dns/dns-default-tf2qd"
Jul 08 16:35:15.560348 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:15.560356    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-registry-tls\") pod \"image-registry-795cc6499b-97mrt\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") " pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:35:15.562623 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:15.562601    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/187d83d2-dbc5-46aa-8e7b-f3df420b474c-metrics-tls\") pod \"dns-default-tf2qd\" (UID: \"187d83d2-dbc5-46aa-8e7b-f3df420b474c\") " pod="openshift-dns/dns-default-tf2qd"
Jul 08 16:35:15.562763 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:15.562696    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-registry-tls\") pod \"image-registry-795cc6499b-97mrt\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") " pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:35:15.577784 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:15.577760    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:35:15.584834 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:15.584805    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tf2qd"
Jul 08 16:35:15.723898 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:15.723863    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:15.778174 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:15.778143    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-795cc6499b-97mrt"]
Jul 08 16:35:15.781624 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:35:15.781589    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0483b06a_6fc8_4e43_a1f8_99a63e9e61cb.slice/crio-51f255a50ac4c5e1b2eb371b3bff999e50b27de698508b78c4d583bc28a0603b WatchSource:0}: Error finding container 51f255a50ac4c5e1b2eb371b3bff999e50b27de698508b78c4d583bc28a0603b: Status 404 returned error can't find the container with id 51f255a50ac4c5e1b2eb371b3bff999e50b27de698508b78c4d583bc28a0603b
Jul 08 16:35:15.786061 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:15.786038    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tf2qd"]
Jul 08 16:35:15.788670 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:35:15.788649    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod187d83d2_dbc5_46aa_8e7b_f3df420b474c.slice/crio-65c063ae9727210d3aa52a12d2d21d1830cede04558fffd65f526b2753bb43e1 WatchSource:0}: Error finding container 65c063ae9727210d3aa52a12d2d21d1830cede04558fffd65f526b2753bb43e1: Status 404 returned error can't find the container with id 65c063ae9727210d3aa52a12d2d21d1830cede04558fffd65f526b2753bb43e1
Jul 08 16:35:15.925588 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:15.925564    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:16.040942 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:16.040898    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:16.146469 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:16.146368    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:16.180038 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:16.180000    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tf2qd" event={"ID":"187d83d2-dbc5-46aa-8e7b-f3df420b474c","Type":"ContainerStarted","Data":"65c063ae9727210d3aa52a12d2d21d1830cede04558fffd65f526b2753bb43e1"}
Jul 08 16:35:16.181668 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:16.181642    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-795cc6499b-97mrt" event={"ID":"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb","Type":"ContainerStarted","Data":"88410f344f16839a25e8d5e5352227d7c9759ee0dd16d43a87e557bf885d582d"}
Jul 08 16:35:16.181838 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:16.181707    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-795cc6499b-97mrt" event={"ID":"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb","Type":"ContainerStarted","Data":"51f255a50ac4c5e1b2eb371b3bff999e50b27de698508b78c4d583bc28a0603b"}
Jul 08 16:35:16.181838 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:16.181734    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:35:16.215720 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:16.215642    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-795cc6499b-97mrt" podStartSLOduration=20.215616407 podStartE2EDuration="20.215616407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:16.214565217 +0000 UTC m=+52.927926644" watchObservedRunningTime="2026-07-08 16:35:16.215616407 +0000 UTC m=+52.928977836"
Jul 08 16:35:16.414170 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:16.414087    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:16.526282 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:16.526249    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:17.433444 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:17.433406    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:17.522262 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:17.522205    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:17.716544 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:17.716444    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:18.144017 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:18.143928    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:18.322287 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:18.322261    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:18.730765 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:18.730727    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:19.190417 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:19.190384    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tf2qd" event={"ID":"187d83d2-dbc5-46aa-8e7b-f3df420b474c","Type":"ContainerStarted","Data":"6f023bb9b0d7a300b7dabcbb8fea8504aa6440f55c004bd0b049c0d1fad6e76c"}
Jul 08 16:35:19.190580 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:19.190465    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tf2qd" event={"ID":"187d83d2-dbc5-46aa-8e7b-f3df420b474c","Type":"ContainerStarted","Data":"79e209474d7198c1dafa21dfbe41c5ac7ae8200f7c3f199da354b641c1f81d4e"}
Jul 08 16:35:19.190580 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:19.190506    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tf2qd"
Jul 08 16:35:19.278246 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:19.278204    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:19.360416 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:19.360379    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:19.395376 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:19.395336    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59fa9d8c-d7a1-430d-bd58-560305b06ec9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24ccq\" (UID: \"59fa9d8c-d7a1-430d-bd58-560305b06ec9\") " pod="openshift-insights/insights-runtime-extractor-24ccq"
Jul 08 16:35:19.395497 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:19.395440    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59fa9d8c-d7a1-430d-bd58-560305b06ec9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24ccq\" (UID: \"59fa9d8c-d7a1-430d-bd58-560305b06ec9\") " pod="openshift-insights/insights-runtime-extractor-24ccq"
Jul 08 16:35:19.395550 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:19.395498    2565 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:19.395613 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:19.395569    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59fa9d8c-d7a1-430d-bd58-560305b06ec9-insights-runtime-extractor-tls podName:59fa9d8c-d7a1-430d-bd58-560305b06ec9 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:35.395549334 +0000 UTC m=+72.108910752 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/59fa9d8c-d7a1-430d-bd58-560305b06ec9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-24ccq" (UID: "59fa9d8c-d7a1-430d-bd58-560305b06ec9") : secret "insights-runtime-extractor-tls" not found
Jul 08 16:35:19.395985 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:19.395968    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/59fa9d8c-d7a1-430d-bd58-560305b06ec9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24ccq\" (UID: \"59fa9d8c-d7a1-430d-bd58-560305b06ec9\") " pod="openshift-insights/insights-runtime-extractor-24ccq"
Jul 08 16:35:19.529572 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:19.529488    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:20.142048 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:20.142013    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:20.388610 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:20.388569    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:20.528223 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:20.528189    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:22.139032 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:22.138996    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:22.216630 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:22.216598    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:22.502506 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:22.502477    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:22.609773 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:22.609749    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:22.730095 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:22.730055    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:22.915818 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:22.915734    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:22.997454 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:22.997427    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:23.106739 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:23.106705    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:23.523078 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:23.523045    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:24.140330 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:24.140289    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:24.325553 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:24.325520    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:24.480086 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:24.480056    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:24.579148 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:24.579119    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:26.120120 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:26.120091    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gb8xd"
Jul 08 16:35:26.153105 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:26.152618    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tf2qd" podStartSLOduration=24.712654258 podStartE2EDuration="27.152595544s" totalImagesPullingTime="2.439941286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:19.216103711 +0000 UTC m=+55.929465139" watchObservedRunningTime="2026-07-08 16:35:26.152595544 +0000 UTC m=+62.865956974"
Jul 08 16:35:29.195666 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:29.195635    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tf2qd"
Jul 08 16:35:29.474273 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:29.474169    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14d465c-2bea-496d-8112-450e25ae7717-metrics-certs\") pod \"network-metrics-daemon-bxjqc\" (UID: \"d14d465c-2bea-496d-8112-450e25ae7717\") " pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:35:29.477221 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:29.477205    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 08 16:35:29.486888 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:29.486856    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d14d465c-2bea-496d-8112-450e25ae7717-metrics-certs\") pod \"network-metrics-daemon-bxjqc\" (UID: \"d14d465c-2bea-496d-8112-450e25ae7717\") " pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:35:29.574865 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:29.574818    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-original-pull-secret\") pod \"global-pull-secret-syncer-sxzrc\" (UID: \"3cf84d3d-0028-4d7f-b178-c4a0d8391cdc\") " pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:35:29.574865 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:29.574877    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5zv5\" (UniqueName: \"kubernetes.io/projected/dc585660-f572-4e9e-80b3-63e57d32f369-kube-api-access-w5zv5\") pod \"network-check-target-8m52j\" (UID: \"dc585660-f572-4e9e-80b3-63e57d32f369\") " pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:35:29.578372 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:29.578352    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 08 16:35:29.578820 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:29.578806    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 08 16:35:29.587282 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:29.587251    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3cf84d3d-0028-4d7f-b178-c4a0d8391cdc-original-pull-secret\") pod \"global-pull-secret-syncer-sxzrc\" (UID: \"3cf84d3d-0028-4d7f-b178-c4a0d8391cdc\") " pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:35:29.588151 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:29.588131    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 08 16:35:29.597981 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:29.597950    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5zv5\" (UniqueName: \"kubernetes.io/projected/dc585660-f572-4e9e-80b3-63e57d32f369-kube-api-access-w5zv5\") pod \"network-check-target-8m52j\" (UID: \"dc585660-f572-4e9e-80b3-63e57d32f369\") " pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:35:29.631809 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:29.631786    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-zs6bx\""
Jul 08 16:35:29.638160 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:29.638143    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-g78k4\""
Jul 08 16:35:29.639200 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:29.639185    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:35:29.643094 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:29.643078    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-sxzrc"
Jul 08 16:35:29.646573 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:29.646551    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bxjqc"
Jul 08 16:35:29.793586 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:29.793557    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bxjqc"]
Jul 08 16:35:29.796373 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:35:29.796340    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd14d465c_2bea_496d_8112_450e25ae7717.slice/crio-58e6c094b627718c9edacbaea9dc5111718cdf98ace83e6eb278bfb60a9c1c15 WatchSource:0}: Error finding container 58e6c094b627718c9edacbaea9dc5111718cdf98ace83e6eb278bfb60a9c1c15: Status 404 returned error can't find the container with id 58e6c094b627718c9edacbaea9dc5111718cdf98ace83e6eb278bfb60a9c1c15
Jul 08 16:35:30.007933 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.007856    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-sxzrc"]
Jul 08 16:35:30.011741 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.011719    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8m52j"]
Jul 08 16:35:30.011832 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:35:30.011739    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf84d3d_0028_4d7f_b178_c4a0d8391cdc.slice/crio-d1b887f8cad2941ee95d970e881eb6262ccadfa4e28b12427d3003ce667f0984 WatchSource:0}: Error finding container d1b887f8cad2941ee95d970e881eb6262ccadfa4e28b12427d3003ce667f0984: Status 404 returned error can't find the container with id d1b887f8cad2941ee95d970e881eb6262ccadfa4e28b12427d3003ce667f0984
Jul 08 16:35:30.015599 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:35:30.015578    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc585660_f572_4e9e_80b3_63e57d32f369.slice/crio-55159cf417a8ebcf1b4adab0ff9d23fd6c0dabcda466a0032a3d33f54156dbd4 WatchSource:0}: Error finding container 55159cf417a8ebcf1b4adab0ff9d23fd6c0dabcda466a0032a3d33f54156dbd4: Status 404 returned error can't find the container with id 55159cf417a8ebcf1b4adab0ff9d23fd6c0dabcda466a0032a3d33f54156dbd4
Jul 08 16:35:30.221627 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.221595    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sxzrc" event={"ID":"3cf84d3d-0028-4d7f-b178-c4a0d8391cdc","Type":"ContainerStarted","Data":"d1b887f8cad2941ee95d970e881eb6262ccadfa4e28b12427d3003ce667f0984"}
Jul 08 16:35:30.222723 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.222697    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8m52j" event={"ID":"dc585660-f572-4e9e-80b3-63e57d32f369","Type":"ContainerStarted","Data":"55159cf417a8ebcf1b4adab0ff9d23fd6c0dabcda466a0032a3d33f54156dbd4"}
Jul 08 16:35:30.223779 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.223757    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bxjqc" event={"ID":"d14d465c-2bea-496d-8112-450e25ae7717","Type":"ContainerStarted","Data":"58e6c094b627718c9edacbaea9dc5111718cdf98ace83e6eb278bfb60a9c1c15"}
Jul 08 16:35:30.551892 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.551758    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c6466d5b6-tlm5j"]
Jul 08 16:35:30.554598 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.554574    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c6466d5b6-tlm5j"
Jul 08 16:35:30.560876 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.560844    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 08 16:35:30.561547 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.561523    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 08 16:35:30.561656 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.561535    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-hnggr\""
Jul 08 16:35:30.561957 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.561911    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6d6f45d4-bjtb6"]
Jul 08 16:35:30.564449 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.564428    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 08 16:35:30.564560 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.564477    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 08 16:35:30.564657 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.564642    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6d6f45d4-bjtb6"
Jul 08 16:35:30.567420 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.567399    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 08 16:35:30.575303 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.574534    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c6466d5b6-tlm5j"]
Jul 08 16:35:30.582147 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.582070    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6d6f45d4-bjtb6"]
Jul 08 16:35:30.583655 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.583632    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-795cc6499b-97mrt"]
Jul 08 16:35:30.589939 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.589910    2565 patch_prober.go:28] interesting pod/image-registry-795cc6499b-97mrt container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 08 16:35:30.590045 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.589965    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-795cc6499b-97mrt" podUID="0483b06a-6fc8-4e43-a1f8-99a63e9e61cb" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:35:30.683318 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.683278    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8278ec4-3d07-44ed-906c-a3bc97d86c5a-tmp\") pod \"klusterlet-addon-workmgr-5c6d6f45d4-bjtb6\" (UID: \"c8278ec4-3d07-44ed-906c-a3bc97d86c5a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6d6f45d4-bjtb6"
Jul 08 16:35:30.683578 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.683345    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c8278ec4-3d07-44ed-906c-a3bc97d86c5a-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c6d6f45d4-bjtb6\" (UID: \"c8278ec4-3d07-44ed-906c-a3bc97d86c5a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6d6f45d4-bjtb6"
Jul 08 16:35:30.683578 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.683388    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4dd9\" (UniqueName: \"kubernetes.io/projected/c8278ec4-3d07-44ed-906c-a3bc97d86c5a-kube-api-access-w4dd9\") pod \"klusterlet-addon-workmgr-5c6d6f45d4-bjtb6\" (UID: \"c8278ec4-3d07-44ed-906c-a3bc97d86c5a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6d6f45d4-bjtb6"
Jul 08 16:35:30.683578 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.683411    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn7f8\" (UniqueName: \"kubernetes.io/projected/dba4bc6b-c747-45ea-9d31-1232c1c7607e-kube-api-access-vn7f8\") pod \"managed-serviceaccount-addon-agent-5c6466d5b6-tlm5j\" (UID: \"dba4bc6b-c747-45ea-9d31-1232c1c7607e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c6466d5b6-tlm5j"
Jul 08 16:35:30.683578 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.683470    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dba4bc6b-c747-45ea-9d31-1232c1c7607e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5c6466d5b6-tlm5j\" (UID: \"dba4bc6b-c747-45ea-9d31-1232c1c7607e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c6466d5b6-tlm5j"
Jul 08 16:35:30.693067 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.693026    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:30.784777 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.784735    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8278ec4-3d07-44ed-906c-a3bc97d86c5a-tmp\") pod \"klusterlet-addon-workmgr-5c6d6f45d4-bjtb6\" (UID: \"c8278ec4-3d07-44ed-906c-a3bc97d86c5a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6d6f45d4-bjtb6"
Jul 08 16:35:30.784942 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.784821    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c8278ec4-3d07-44ed-906c-a3bc97d86c5a-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c6d6f45d4-bjtb6\" (UID: \"c8278ec4-3d07-44ed-906c-a3bc97d86c5a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6d6f45d4-bjtb6"
Jul 08 16:35:30.784942 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.784872    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4dd9\" (UniqueName: \"kubernetes.io/projected/c8278ec4-3d07-44ed-906c-a3bc97d86c5a-kube-api-access-w4dd9\") pod \"klusterlet-addon-workmgr-5c6d6f45d4-bjtb6\" (UID: \"c8278ec4-3d07-44ed-906c-a3bc97d86c5a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6d6f45d4-bjtb6"
Jul 08 16:35:30.785115 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.785085    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn7f8\" (UniqueName: \"kubernetes.io/projected/dba4bc6b-c747-45ea-9d31-1232c1c7607e-kube-api-access-vn7f8\") pod \"managed-serviceaccount-addon-agent-5c6466d5b6-tlm5j\" (UID: \"dba4bc6b-c747-45ea-9d31-1232c1c7607e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c6466d5b6-tlm5j"
Jul 08 16:35:30.785192 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.785142    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dba4bc6b-c747-45ea-9d31-1232c1c7607e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5c6466d5b6-tlm5j\" (UID: \"dba4bc6b-c747-45ea-9d31-1232c1c7607e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c6466d5b6-tlm5j"
Jul 08 16:35:30.785192 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.785172    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8278ec4-3d07-44ed-906c-a3bc97d86c5a-tmp\") pod \"klusterlet-addon-workmgr-5c6d6f45d4-bjtb6\" (UID: \"c8278ec4-3d07-44ed-906c-a3bc97d86c5a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6d6f45d4-bjtb6"
Jul 08 16:35:30.787928 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.787902    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c8278ec4-3d07-44ed-906c-a3bc97d86c5a-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c6d6f45d4-bjtb6\" (UID: \"c8278ec4-3d07-44ed-906c-a3bc97d86c5a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6d6f45d4-bjtb6"
Jul 08 16:35:30.788496 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.788473    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dba4bc6b-c747-45ea-9d31-1232c1c7607e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5c6466d5b6-tlm5j\" (UID: \"dba4bc6b-c747-45ea-9d31-1232c1c7607e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c6466d5b6-tlm5j"
Jul 08 16:35:30.796388 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.796362    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn7f8\" (UniqueName: \"kubernetes.io/projected/dba4bc6b-c747-45ea-9d31-1232c1c7607e-kube-api-access-vn7f8\") pod \"managed-serviceaccount-addon-agent-5c6466d5b6-tlm5j\" (UID: \"dba4bc6b-c747-45ea-9d31-1232c1c7607e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c6466d5b6-tlm5j"
Jul 08 16:35:30.796977 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.796949    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4dd9\" (UniqueName: \"kubernetes.io/projected/c8278ec4-3d07-44ed-906c-a3bc97d86c5a-kube-api-access-w4dd9\") pod \"klusterlet-addon-workmgr-5c6d6f45d4-bjtb6\" (UID: \"c8278ec4-3d07-44ed-906c-a3bc97d86c5a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6d6f45d4-bjtb6"
Jul 08 16:35:30.797564 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.797513    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:30.874637 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.874561    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c6466d5b6-tlm5j"
Jul 08 16:35:30.884894 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.884865    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6d6f45d4-bjtb6"
Jul 08 16:35:30.896355 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:30.896324    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:31.007215 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.007152    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:31.053866 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.053841    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6d6f45d4-bjtb6"]
Jul 08 16:35:31.057759 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:35:31.057723    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8278ec4_3d07_44ed_906c_a3bc97d86c5a.slice/crio-86a7eed5959658bc30e9e31948e209e9f23e52cbdadfd9c584ff2b9406d69a1f WatchSource:0}: Error finding container 86a7eed5959658bc30e9e31948e209e9f23e52cbdadfd9c584ff2b9406d69a1f: Status 404 returned error can't find the container with id 86a7eed5959658bc30e9e31948e209e9f23e52cbdadfd9c584ff2b9406d69a1f
Jul 08 16:35:31.068503 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.068475    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c6466d5b6-tlm5j"]
Jul 08 16:35:31.071986 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:35:31.071961    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddba4bc6b_c747_45ea_9d31_1232c1c7607e.slice/crio-560fda3735dcf41ad2b6106a36f3520194ee6ba7c742c4ac194bcd01b2cf658c WatchSource:0}: Error finding container 560fda3735dcf41ad2b6106a36f3520194ee6ba7c742c4ac194bcd01b2cf658c: Status 404 returned error can't find the container with id 560fda3735dcf41ad2b6106a36f3520194ee6ba7c742c4ac194bcd01b2cf658c
Jul 08 16:35:31.104892 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.104702    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:31.233477 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.233378    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6d6f45d4-bjtb6" event={"ID":"c8278ec4-3d07-44ed-906c-a3bc97d86c5a","Type":"ContainerStarted","Data":"86a7eed5959658bc30e9e31948e209e9f23e52cbdadfd9c584ff2b9406d69a1f"}
Jul 08 16:35:31.237775 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.237732    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c6466d5b6-tlm5j" event={"ID":"dba4bc6b-c747-45ea-9d31-1232c1c7607e","Type":"ContainerStarted","Data":"560fda3735dcf41ad2b6106a36f3520194ee6ba7c742c4ac194bcd01b2cf658c"}
Jul 08 16:35:31.351757 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.351724    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59d4bb4df4-qjlqg"]
Jul 08 16:35:31.358812 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.358788    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d4bb4df4-qjlqg"
Jul 08 16:35:31.361952 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.361639    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 08 16:35:31.361952 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.361786    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 08 16:35:31.361952 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.361911    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 08 16:35:31.362257 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.362005    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 08 16:35:31.362945 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.362770    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kfjdt\""
Jul 08 16:35:31.363056 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.362995    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 08 16:35:31.363246 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.363229    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 08 16:35:31.363339 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.363255    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 08 16:35:31.371267 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.371211    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59d4bb4df4-qjlqg"]
Jul 08 16:35:31.491638 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.491540    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fbd600c-a512-46e6-af5f-c09771373fe9-oauth-serving-cert\") pod \"console-59d4bb4df4-qjlqg\" (UID: \"9fbd600c-a512-46e6-af5f-c09771373fe9\") " pod="openshift-console/console-59d4bb4df4-qjlqg"
Jul 08 16:35:31.491638 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.491626    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fbd600c-a512-46e6-af5f-c09771373fe9-service-ca\") pod \"console-59d4bb4df4-qjlqg\" (UID: \"9fbd600c-a512-46e6-af5f-c09771373fe9\") " pod="openshift-console/console-59d4bb4df4-qjlqg"
Jul 08 16:35:31.491882 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.491709    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fbd600c-a512-46e6-af5f-c09771373fe9-console-serving-cert\") pod \"console-59d4bb4df4-qjlqg\" (UID: \"9fbd600c-a512-46e6-af5f-c09771373fe9\") " pod="openshift-console/console-59d4bb4df4-qjlqg"
Jul 08 16:35:31.491938 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.491913    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fbd600c-a512-46e6-af5f-c09771373fe9-console-oauth-config\") pod \"console-59d4bb4df4-qjlqg\" (UID: \"9fbd600c-a512-46e6-af5f-c09771373fe9\") " pod="openshift-console/console-59d4bb4df4-qjlqg"
Jul 08 16:35:31.492048 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.491979    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fbd600c-a512-46e6-af5f-c09771373fe9-console-config\") pod \"console-59d4bb4df4-qjlqg\" (UID: \"9fbd600c-a512-46e6-af5f-c09771373fe9\") " pod="openshift-console/console-59d4bb4df4-qjlqg"
Jul 08 16:35:31.492138 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.492066    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96mt6\" (UniqueName: \"kubernetes.io/projected/9fbd600c-a512-46e6-af5f-c09771373fe9-kube-api-access-96mt6\") pod \"console-59d4bb4df4-qjlqg\" (UID: \"9fbd600c-a512-46e6-af5f-c09771373fe9\") " pod="openshift-console/console-59d4bb4df4-qjlqg"
Jul 08 16:35:31.593384 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.593024    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fbd600c-a512-46e6-af5f-c09771373fe9-console-serving-cert\") pod \"console-59d4bb4df4-qjlqg\" (UID: \"9fbd600c-a512-46e6-af5f-c09771373fe9\") " pod="openshift-console/console-59d4bb4df4-qjlqg"
Jul 08 16:35:31.593384 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.593078    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fbd600c-a512-46e6-af5f-c09771373fe9-console-oauth-config\") pod \"console-59d4bb4df4-qjlqg\" (UID: \"9fbd600c-a512-46e6-af5f-c09771373fe9\") " pod="openshift-console/console-59d4bb4df4-qjlqg"
Jul 08 16:35:31.593384 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.593131    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fbd600c-a512-46e6-af5f-c09771373fe9-console-config\") pod \"console-59d4bb4df4-qjlqg\" (UID: \"9fbd600c-a512-46e6-af5f-c09771373fe9\") " pod="openshift-console/console-59d4bb4df4-qjlqg"
Jul 08 16:35:31.593384 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.593172    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96mt6\" (UniqueName: \"kubernetes.io/projected/9fbd600c-a512-46e6-af5f-c09771373fe9-kube-api-access-96mt6\") pod \"console-59d4bb4df4-qjlqg\" (UID: \"9fbd600c-a512-46e6-af5f-c09771373fe9\") " pod="openshift-console/console-59d4bb4df4-qjlqg"
Jul 08 16:35:31.593384 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.593205    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fbd600c-a512-46e6-af5f-c09771373fe9-oauth-serving-cert\") pod \"console-59d4bb4df4-qjlqg\" (UID: \"9fbd600c-a512-46e6-af5f-c09771373fe9\") " pod="openshift-console/console-59d4bb4df4-qjlqg"
Jul 08 16:35:31.593384 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.593238    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fbd600c-a512-46e6-af5f-c09771373fe9-service-ca\") pod \"console-59d4bb4df4-qjlqg\" (UID: \"9fbd600c-a512-46e6-af5f-c09771373fe9\") " pod="openshift-console/console-59d4bb4df4-qjlqg"
Jul 08 16:35:31.594102 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.594073    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fbd600c-a512-46e6-af5f-c09771373fe9-oauth-serving-cert\") pod \"console-59d4bb4df4-qjlqg\" (UID: \"9fbd600c-a512-46e6-af5f-c09771373fe9\") " pod="openshift-console/console-59d4bb4df4-qjlqg"
Jul 08 16:35:31.594263 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.594232    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fbd600c-a512-46e6-af5f-c09771373fe9-service-ca\") pod \"console-59d4bb4df4-qjlqg\" (UID: \"9fbd600c-a512-46e6-af5f-c09771373fe9\") " pod="openshift-console/console-59d4bb4df4-qjlqg"
Jul 08 16:35:31.594521 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.594481    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fbd600c-a512-46e6-af5f-c09771373fe9-console-config\") pod \"console-59d4bb4df4-qjlqg\" (UID: \"9fbd600c-a512-46e6-af5f-c09771373fe9\") " pod="openshift-console/console-59d4bb4df4-qjlqg"
Jul 08 16:35:31.596751 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.596729    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fbd600c-a512-46e6-af5f-c09771373fe9-console-serving-cert\") pod \"console-59d4bb4df4-qjlqg\" (UID: \"9fbd600c-a512-46e6-af5f-c09771373fe9\") " pod="openshift-console/console-59d4bb4df4-qjlqg"
Jul 08 16:35:31.596849 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.596761    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fbd600c-a512-46e6-af5f-c09771373fe9-console-oauth-config\") pod \"console-59d4bb4df4-qjlqg\" (UID: \"9fbd600c-a512-46e6-af5f-c09771373fe9\") " pod="openshift-console/console-59d4bb4df4-qjlqg"
Jul 08 16:35:31.599079 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.598916    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:31.606351 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.606298    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96mt6\" (UniqueName: \"kubernetes.io/projected/9fbd600c-a512-46e6-af5f-c09771373fe9-kube-api-access-96mt6\") pod \"console-59d4bb4df4-qjlqg\" (UID: \"9fbd600c-a512-46e6-af5f-c09771373fe9\") " pod="openshift-console/console-59d4bb4df4-qjlqg"
Jul 08 16:35:31.673201 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.673164    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d4bb4df4-qjlqg"
Jul 08 16:35:31.820540 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.820502    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:31.864910 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:31.864875    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59d4bb4df4-qjlqg"]
Jul 08 16:35:31.993578 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:35:31.993537    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fbd600c_a512_46e6_af5f_c09771373fe9.slice/crio-81f6d61da220b8d4e7428507aaa6d9973589ffe60aefa7d7b8684189e4d88770 WatchSource:0}: Error finding container 81f6d61da220b8d4e7428507aaa6d9973589ffe60aefa7d7b8684189e4d88770: Status 404 returned error can't find the container with id 81f6d61da220b8d4e7428507aaa6d9973589ffe60aefa7d7b8684189e4d88770
Jul 08 16:35:32.248555 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:32.248472    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d4bb4df4-qjlqg" event={"ID":"9fbd600c-a512-46e6-af5f-c09771373fe9","Type":"ContainerStarted","Data":"81f6d61da220b8d4e7428507aaa6d9973589ffe60aefa7d7b8684189e4d88770"}
Jul 08 16:35:32.344882 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:32.343880    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:32.439184 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:32.439147    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:32.515917 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:32.515864    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:33.262648 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:33.261819    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bxjqc" event={"ID":"d14d465c-2bea-496d-8112-450e25ae7717","Type":"ContainerStarted","Data":"eacfb7d95b5ca6144f2434d9de3bccdbbd1eda210a90f2c827bec8057aa1d988"}
Jul 08 16:35:33.262648 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:33.261888    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bxjqc" event={"ID":"d14d465c-2bea-496d-8112-450e25ae7717","Type":"ContainerStarted","Data":"f62a507b1a89e6d9ef5e0abd0335a194e2a90de653fe0e3e36ca4a9528933c24"}
Jul 08 16:35:33.284143 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:33.283377    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bxjqc" podStartSLOduration=67.048892441 podStartE2EDuration="1m9.283353409s" totalImagesPullingTime="2.234460968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:33.282204697 +0000 UTC m=+69.995566160" watchObservedRunningTime="2026-07-08 16:35:33.283353409 +0000 UTC m=+69.996714837"
Jul 08 16:35:33.422879 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:33.422829    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:33.485098 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:33.485032    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:33.552284 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:33.552203    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:33.787060 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:33.787013    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:34.029821 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:34.029778    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:34.104550 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:34.103894    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:34.946165 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:34.946125    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:35.045826 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:35.045789    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:35.432002 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:35.431968    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59fa9d8c-d7a1-430d-bd58-560305b06ec9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24ccq\" (UID: \"59fa9d8c-d7a1-430d-bd58-560305b06ec9\") " pod="openshift-insights/insights-runtime-extractor-24ccq"
Jul 08 16:35:35.434167 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:35.434145    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/59fa9d8c-d7a1-430d-bd58-560305b06ec9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24ccq\" (UID: \"59fa9d8c-d7a1-430d-bd58-560305b06ec9\") " pod="openshift-insights/insights-runtime-extractor-24ccq"
Jul 08 16:35:35.619735 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:35.619700    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cshmv\""
Jul 08 16:35:35.628142 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:35.628109    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-24ccq"
Jul 08 16:35:35.846781 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:35.846694    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:35.936609 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:35.936557    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:36.442902 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:36.442863    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:37.103561 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:37.103515    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:37.236194 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:37.236161    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:39.042155 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:39.042092    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:39.123305 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:39.123265    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:39.518233 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:39.518202    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-24ccq"]
Jul 08 16:35:39.646584 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:39.646498    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:39.747150 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:39.747117    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:40.293118 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:40.293084    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6d6f45d4-bjtb6" event={"ID":"c8278ec4-3d07-44ed-906c-a3bc97d86c5a","Type":"ContainerStarted","Data":"e7460497aff16d08084f86f6502629a6eda371093d1813641473800cad28be05"}
Jul 08 16:35:40.293590 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:40.293264    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6d6f45d4-bjtb6"
Jul 08 16:35:40.295734 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:40.295128    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d4bb4df4-qjlqg" event={"ID":"9fbd600c-a512-46e6-af5f-c09771373fe9","Type":"ContainerStarted","Data":"827789282b2907f15bb5e6e62f2c134dc1321660ea284aef7fe1fc695f80ac7f"}
Jul 08 16:35:40.295734 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:40.295689    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6d6f45d4-bjtb6"
Jul 08 16:35:40.296465 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:40.296440    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-24ccq" event={"ID":"59fa9d8c-d7a1-430d-bd58-560305b06ec9","Type":"ContainerStarted","Data":"a0c76e6468108a4108169a2224f2c9c28b7c8f939096dea2d82b6f9e8d17aaf9"}
Jul 08 16:35:40.299382 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:40.299332    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-sxzrc" event={"ID":"3cf84d3d-0028-4d7f-b178-c4a0d8391cdc","Type":"ContainerStarted","Data":"74d1be9b9e7100ce37280a6120b911918efc153ec0b5a2c7458c4e19c84d9138"}
Jul 08 16:35:40.301219 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:40.301179    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8m52j" event={"ID":"dc585660-f572-4e9e-80b3-63e57d32f369","Type":"ContainerStarted","Data":"a970d2c1a6fa4bb2fd22d756143bf13d21e12e70ecef64537928661915425e91"}
Jul 08 16:35:40.301637 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:40.301587    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:35:40.303593 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:40.303570    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c6466d5b6-tlm5j" event={"ID":"dba4bc6b-c747-45ea-9d31-1232c1c7607e","Type":"ContainerStarted","Data":"b24e1eae9a0c820d5a2ca8917922ce4d063e524c2087c27ca72e7737ec8f1085"}
Jul 08 16:35:40.314123 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:40.314071    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c6d6f45d4-bjtb6" podStartSLOduration=2.003658437 podStartE2EDuration="10.314050876s" totalImagesPullingTime="8.310392439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:40.313144606 +0000 UTC m=+77.026506037" watchObservedRunningTime="2026-07-08 16:35:40.314050876 +0000 UTC m=+77.027412304"
Jul 08 16:35:40.331493 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:40.331437    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-sxzrc" podStartSLOduration=67.00516543 podStartE2EDuration="1m16.33141727s" totalImagesPullingTime="9.32625184s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:40.331020029 +0000 UTC m=+77.044381478" watchObservedRunningTime="2026-07-08 16:35:40.33141727 +0000 UTC m=+77.044778698"
Jul 08 16:35:40.348767 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:40.348714    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8m52j" podStartSLOduration=67.024228459 podStartE2EDuration="1m16.348697822s" totalImagesPullingTime="9.324469363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:40.348308514 +0000 UTC m=+77.061669943" watchObservedRunningTime="2026-07-08 16:35:40.348697822 +0000 UTC m=+77.062059242"
Jul 08 16:35:40.367714 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:40.367482    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5c6466d5b6-tlm5j" podStartSLOduration=2.087614094 podStartE2EDuration="10.367462461s" totalImagesPullingTime="8.279848367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:40.367217917 +0000 UTC m=+77.080579345" watchObservedRunningTime="2026-07-08 16:35:40.367462461 +0000 UTC m=+77.080823890"
Jul 08 16:35:40.390837 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:40.390780    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59d4bb4df4-qjlqg" podStartSLOduration=2.073263767 podStartE2EDuration="9.390760434s" totalImagesPullingTime="7.317496667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:40.389966972 +0000 UTC m=+77.103328426" watchObservedRunningTime="2026-07-08 16:35:40.390760434 +0000 UTC m=+77.104121865"
Jul 08 16:35:40.590987 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:40.590876    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:35:40.778735 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:40.778697    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:40.840188 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:40.840150    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:41.520639 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.520600    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:41.536953 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.536898    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-559cc97d7b-z4pnc"]
Jul 08 16:35:41.547287 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.547261    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-559cc97d7b-z4pnc"
Jul 08 16:35:41.553319 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.553297    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-559cc97d7b-z4pnc"]
Jul 08 16:35:41.556197 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.556143    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 08 16:35:41.673763 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.673735    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59d4bb4df4-qjlqg"
Jul 08 16:35:41.673937 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.673811    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59d4bb4df4-qjlqg"
Jul 08 16:35:41.679403 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.679380    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59d4bb4df4-qjlqg"
Jul 08 16:35:41.687840 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.687812    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af53fddb-4cb2-42c7-a274-6b74101a0f0e-service-ca\") pod \"console-559cc97d7b-z4pnc\" (UID: \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\") " pod="openshift-console/console-559cc97d7b-z4pnc"
Jul 08 16:35:41.687978 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.687860    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af53fddb-4cb2-42c7-a274-6b74101a0f0e-console-oauth-config\") pod \"console-559cc97d7b-z4pnc\" (UID: \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\") " pod="openshift-console/console-559cc97d7b-z4pnc"
Jul 08 16:35:41.687978 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.687892    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af53fddb-4cb2-42c7-a274-6b74101a0f0e-trusted-ca-bundle\") pod \"console-559cc97d7b-z4pnc\" (UID: \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\") " pod="openshift-console/console-559cc97d7b-z4pnc"
Jul 08 16:35:41.687978 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.687928    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl9ds\" (UniqueName: \"kubernetes.io/projected/af53fddb-4cb2-42c7-a274-6b74101a0f0e-kube-api-access-hl9ds\") pod \"console-559cc97d7b-z4pnc\" (UID: \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\") " pod="openshift-console/console-559cc97d7b-z4pnc"
Jul 08 16:35:41.688117 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.687983    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af53fddb-4cb2-42c7-a274-6b74101a0f0e-console-serving-cert\") pod \"console-559cc97d7b-z4pnc\" (UID: \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\") " pod="openshift-console/console-559cc97d7b-z4pnc"
Jul 08 16:35:41.688117 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.688011    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af53fddb-4cb2-42c7-a274-6b74101a0f0e-console-config\") pod \"console-559cc97d7b-z4pnc\" (UID: \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\") " pod="openshift-console/console-559cc97d7b-z4pnc"
Jul 08 16:35:41.688117 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.688029    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af53fddb-4cb2-42c7-a274-6b74101a0f0e-oauth-serving-cert\") pod \"console-559cc97d7b-z4pnc\" (UID: \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\") " pod="openshift-console/console-559cc97d7b-z4pnc"
Jul 08 16:35:41.789284 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.789203    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af53fddb-4cb2-42c7-a274-6b74101a0f0e-console-serving-cert\") pod \"console-559cc97d7b-z4pnc\" (UID: \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\") " pod="openshift-console/console-559cc97d7b-z4pnc"
Jul 08 16:35:41.789284 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.789266    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af53fddb-4cb2-42c7-a274-6b74101a0f0e-console-config\") pod \"console-559cc97d7b-z4pnc\" (UID: \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\") " pod="openshift-console/console-559cc97d7b-z4pnc"
Jul 08 16:35:41.789519 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.789296    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af53fddb-4cb2-42c7-a274-6b74101a0f0e-oauth-serving-cert\") pod \"console-559cc97d7b-z4pnc\" (UID: \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\") " pod="openshift-console/console-559cc97d7b-z4pnc"
Jul 08 16:35:41.789519 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.789344    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af53fddb-4cb2-42c7-a274-6b74101a0f0e-service-ca\") pod \"console-559cc97d7b-z4pnc\" (UID: \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\") " pod="openshift-console/console-559cc97d7b-z4pnc"
Jul 08 16:35:41.789519 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.789394    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af53fddb-4cb2-42c7-a274-6b74101a0f0e-console-oauth-config\") pod \"console-559cc97d7b-z4pnc\" (UID: \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\") " pod="openshift-console/console-559cc97d7b-z4pnc"
Jul 08 16:35:41.789519 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.789422    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af53fddb-4cb2-42c7-a274-6b74101a0f0e-trusted-ca-bundle\") pod \"console-559cc97d7b-z4pnc\" (UID: \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\") " pod="openshift-console/console-559cc97d7b-z4pnc"
Jul 08 16:35:41.789519 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.789449    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl9ds\" (UniqueName: \"kubernetes.io/projected/af53fddb-4cb2-42c7-a274-6b74101a0f0e-kube-api-access-hl9ds\") pod \"console-559cc97d7b-z4pnc\" (UID: \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\") " pod="openshift-console/console-559cc97d7b-z4pnc"
Jul 08 16:35:41.790295 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.790252    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af53fddb-4cb2-42c7-a274-6b74101a0f0e-console-config\") pod \"console-559cc97d7b-z4pnc\" (UID: \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\") " pod="openshift-console/console-559cc97d7b-z4pnc"
Jul 08 16:35:41.790411 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.790399    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af53fddb-4cb2-42c7-a274-6b74101a0f0e-service-ca\") pod \"console-559cc97d7b-z4pnc\" (UID: \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\") " pod="openshift-console/console-559cc97d7b-z4pnc"
Jul 08 16:35:41.790536 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.790508    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af53fddb-4cb2-42c7-a274-6b74101a0f0e-oauth-serving-cert\") pod \"console-559cc97d7b-z4pnc\" (UID: \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\") " pod="openshift-console/console-559cc97d7b-z4pnc"
Jul 08 16:35:41.790643 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.790624    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af53fddb-4cb2-42c7-a274-6b74101a0f0e-trusted-ca-bundle\") pod \"console-559cc97d7b-z4pnc\" (UID: \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\") " pod="openshift-console/console-559cc97d7b-z4pnc"
Jul 08 16:35:41.792211 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.792188    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af53fddb-4cb2-42c7-a274-6b74101a0f0e-console-oauth-config\") pod \"console-559cc97d7b-z4pnc\" (UID: \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\") " pod="openshift-console/console-559cc97d7b-z4pnc"
Jul 08 16:35:41.792355 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.792334    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af53fddb-4cb2-42c7-a274-6b74101a0f0e-console-serving-cert\") pod \"console-559cc97d7b-z4pnc\" (UID: \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\") " pod="openshift-console/console-559cc97d7b-z4pnc"
Jul 08 16:35:41.799658 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.799610    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl9ds\" (UniqueName: \"kubernetes.io/projected/af53fddb-4cb2-42c7-a274-6b74101a0f0e-kube-api-access-hl9ds\") pod \"console-559cc97d7b-z4pnc\" (UID: \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\") " pod="openshift-console/console-559cc97d7b-z4pnc"
Jul 08 16:35:41.859480 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:41.859410    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-559cc97d7b-z4pnc"
Jul 08 16:35:42.289499 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:42.289462    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:42.316963 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:42.316937    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59d4bb4df4-qjlqg"
Jul 08 16:35:42.431762 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:42.431724    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:43.049852 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.049796    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:43.260983 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.260806    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:43.317173 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.316994    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zsdvl"]
Jul 08 16:35:43.322028 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.321722    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b8mjr"]
Jul 08 16:35:43.322754 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.322735    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:43.325594 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.325354    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b8mjr"
Jul 08 16:35:43.325594 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.325372    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 08 16:35:43.325594 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.325415    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dzgdb\""
Jul 08 16:35:43.325594 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.325441    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 08 16:35:43.326024 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.326008    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 08 16:35:43.326024 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.326022    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 08 16:35:43.326364 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.326345    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 08 16:35:43.326503 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.326476    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 08 16:35:43.328087 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.328068    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 08 16:35:43.329462 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.329441    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 08 16:35:43.329558 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.329476    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 08 16:35:43.329924 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.329441    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-fkcb5\""
Jul 08 16:35:43.335439 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.335420    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b8mjr"]
Jul 08 16:35:43.360498 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.360458    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:43.403433 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.403170    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3898e889-6916-443f-9d83-dccd019d6544-node-exporter-wtmp\") pod \"node-exporter-zsdvl\" (UID: \"3898e889-6916-443f-9d83-dccd019d6544\") " pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:43.403433 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.403223    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3898e889-6916-443f-9d83-dccd019d6544-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zsdvl\" (UID: \"3898e889-6916-443f-9d83-dccd019d6544\") " pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:43.403433 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.403256    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/53b2bf60-5387-46eb-b264-e0410b55f8b9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-b8mjr\" (UID: \"53b2bf60-5387-46eb-b264-e0410b55f8b9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b8mjr"
Jul 08 16:35:43.403433 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.403327    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3898e889-6916-443f-9d83-dccd019d6544-node-exporter-textfile\") pod \"node-exporter-zsdvl\" (UID: \"3898e889-6916-443f-9d83-dccd019d6544\") " pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:43.403433 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.403355    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/53b2bf60-5387-46eb-b264-e0410b55f8b9-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-b8mjr\" (UID: \"53b2bf60-5387-46eb-b264-e0410b55f8b9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b8mjr"
Jul 08 16:35:43.403433 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.403400    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53b2bf60-5387-46eb-b264-e0410b55f8b9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-b8mjr\" (UID: \"53b2bf60-5387-46eb-b264-e0410b55f8b9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b8mjr"
Jul 08 16:35:43.403866 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.403499    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3898e889-6916-443f-9d83-dccd019d6544-sys\") pod \"node-exporter-zsdvl\" (UID: \"3898e889-6916-443f-9d83-dccd019d6544\") " pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:43.403866 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.403564    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3898e889-6916-443f-9d83-dccd019d6544-node-exporter-accelerators-collector-config\") pod \"node-exporter-zsdvl\" (UID: \"3898e889-6916-443f-9d83-dccd019d6544\") " pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:43.403866 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.403597    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8sfw\" (UniqueName: \"kubernetes.io/projected/53b2bf60-5387-46eb-b264-e0410b55f8b9-kube-api-access-m8sfw\") pod \"kube-state-metrics-7c9dbb8b7d-b8mjr\" (UID: \"53b2bf60-5387-46eb-b264-e0410b55f8b9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b8mjr"
Jul 08 16:35:43.403866 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.403633    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3898e889-6916-443f-9d83-dccd019d6544-metrics-client-ca\") pod \"node-exporter-zsdvl\" (UID: \"3898e889-6916-443f-9d83-dccd019d6544\") " pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:43.403866 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.403665    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3898e889-6916-443f-9d83-dccd019d6544-node-exporter-tls\") pod \"node-exporter-zsdvl\" (UID: \"3898e889-6916-443f-9d83-dccd019d6544\") " pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:43.403866 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.403728    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/53b2bf60-5387-46eb-b264-e0410b55f8b9-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-b8mjr\" (UID: \"53b2bf60-5387-46eb-b264-e0410b55f8b9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b8mjr"
Jul 08 16:35:43.403866 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.403808    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3898e889-6916-443f-9d83-dccd019d6544-root\") pod \"node-exporter-zsdvl\" (UID: \"3898e889-6916-443f-9d83-dccd019d6544\") " pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:43.403866 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.403846    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p28j\" (UniqueName: \"kubernetes.io/projected/3898e889-6916-443f-9d83-dccd019d6544-kube-api-access-4p28j\") pod \"node-exporter-zsdvl\" (UID: \"3898e889-6916-443f-9d83-dccd019d6544\") " pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:43.404194 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.403904    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53b2bf60-5387-46eb-b264-e0410b55f8b9-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-b8mjr\" (UID: \"53b2bf60-5387-46eb-b264-e0410b55f8b9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b8mjr"
Jul 08 16:35:43.440564 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.440525    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:43.505041 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.504990    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3898e889-6916-443f-9d83-dccd019d6544-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zsdvl\" (UID: \"3898e889-6916-443f-9d83-dccd019d6544\") " pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:43.506033 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.505208    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/53b2bf60-5387-46eb-b264-e0410b55f8b9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-b8mjr\" (UID: \"53b2bf60-5387-46eb-b264-e0410b55f8b9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b8mjr"
Jul 08 16:35:43.506033 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.505266    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3898e889-6916-443f-9d83-dccd019d6544-node-exporter-textfile\") pod \"node-exporter-zsdvl\" (UID: \"3898e889-6916-443f-9d83-dccd019d6544\") " pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:43.506033 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.505291    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/53b2bf60-5387-46eb-b264-e0410b55f8b9-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-b8mjr\" (UID: \"53b2bf60-5387-46eb-b264-e0410b55f8b9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b8mjr"
Jul 08 16:35:43.506033 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.505317    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53b2bf60-5387-46eb-b264-e0410b55f8b9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-b8mjr\" (UID: \"53b2bf60-5387-46eb-b264-e0410b55f8b9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b8mjr"
Jul 08 16:35:43.506033 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.505343    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3898e889-6916-443f-9d83-dccd019d6544-sys\") pod \"node-exporter-zsdvl\" (UID: \"3898e889-6916-443f-9d83-dccd019d6544\") " pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:43.506033 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.505371    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3898e889-6916-443f-9d83-dccd019d6544-node-exporter-accelerators-collector-config\") pod \"node-exporter-zsdvl\" (UID: \"3898e889-6916-443f-9d83-dccd019d6544\") " pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:43.506033 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.505400    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8sfw\" (UniqueName: \"kubernetes.io/projected/53b2bf60-5387-46eb-b264-e0410b55f8b9-kube-api-access-m8sfw\") pod \"kube-state-metrics-7c9dbb8b7d-b8mjr\" (UID: \"53b2bf60-5387-46eb-b264-e0410b55f8b9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b8mjr"
Jul 08 16:35:43.506033 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.505436    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3898e889-6916-443f-9d83-dccd019d6544-metrics-client-ca\") pod \"node-exporter-zsdvl\" (UID: \"3898e889-6916-443f-9d83-dccd019d6544\") " pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:43.506033 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.505472    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3898e889-6916-443f-9d83-dccd019d6544-node-exporter-tls\") pod \"node-exporter-zsdvl\" (UID: \"3898e889-6916-443f-9d83-dccd019d6544\") " pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:43.506033 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.505510    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/53b2bf60-5387-46eb-b264-e0410b55f8b9-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-b8mjr\" (UID: \"53b2bf60-5387-46eb-b264-e0410b55f8b9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b8mjr"
Jul 08 16:35:43.506033 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.505554    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3898e889-6916-443f-9d83-dccd019d6544-root\") pod \"node-exporter-zsdvl\" (UID: \"3898e889-6916-443f-9d83-dccd019d6544\") " pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:43.506033 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.505583    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p28j\" (UniqueName: \"kubernetes.io/projected/3898e889-6916-443f-9d83-dccd019d6544-kube-api-access-4p28j\") pod \"node-exporter-zsdvl\" (UID: \"3898e889-6916-443f-9d83-dccd019d6544\") " pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:43.506033 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.505615    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53b2bf60-5387-46eb-b264-e0410b55f8b9-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-b8mjr\" (UID: \"53b2bf60-5387-46eb-b264-e0410b55f8b9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b8mjr"
Jul 08 16:35:43.506033 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.505702    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3898e889-6916-443f-9d83-dccd019d6544-node-exporter-wtmp\") pod \"node-exporter-zsdvl\" (UID: \"3898e889-6916-443f-9d83-dccd019d6544\") " pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:43.506033 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.505892    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3898e889-6916-443f-9d83-dccd019d6544-node-exporter-wtmp\") pod \"node-exporter-zsdvl\" (UID: \"3898e889-6916-443f-9d83-dccd019d6544\") " pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:43.506033 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.505947    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3898e889-6916-443f-9d83-dccd019d6544-sys\") pod \"node-exporter-zsdvl\" (UID: \"3898e889-6916-443f-9d83-dccd019d6544\") " pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:43.506948 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.506104    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3898e889-6916-443f-9d83-dccd019d6544-node-exporter-textfile\") pod \"node-exporter-zsdvl\" (UID: \"3898e889-6916-443f-9d83-dccd019d6544\") " pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:43.510983 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:43.508514    2565 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 08 16:35:43.510983 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:43.508876    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53b2bf60-5387-46eb-b264-e0410b55f8b9-kube-state-metrics-tls podName:53b2bf60-5387-46eb-b264-e0410b55f8b9 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:44.008853162 +0000 UTC m=+80.722214587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/53b2bf60-5387-46eb-b264-e0410b55f8b9-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-b8mjr" (UID: "53b2bf60-5387-46eb-b264-e0410b55f8b9") : secret "kube-state-metrics-tls" not found
Jul 08 16:35:43.510983 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.509035    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3898e889-6916-443f-9d83-dccd019d6544-node-exporter-accelerators-collector-config\") pod \"node-exporter-zsdvl\" (UID: \"3898e889-6916-443f-9d83-dccd019d6544\") " pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:43.510983 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.509649    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3898e889-6916-443f-9d83-dccd019d6544-root\") pod \"node-exporter-zsdvl\" (UID: \"3898e889-6916-443f-9d83-dccd019d6544\") " pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:43.510983 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.510146    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/53b2bf60-5387-46eb-b264-e0410b55f8b9-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-b8mjr\" (UID: \"53b2bf60-5387-46eb-b264-e0410b55f8b9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b8mjr"
Jul 08 16:35:43.510983 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:43.510559    2565 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 08 16:35:43.513506 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.512832    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/53b2bf60-5387-46eb-b264-e0410b55f8b9-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-b8mjr\" (UID: \"53b2bf60-5387-46eb-b264-e0410b55f8b9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b8mjr"
Jul 08 16:35:43.513506 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:43.513098    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3898e889-6916-443f-9d83-dccd019d6544-node-exporter-tls podName:3898e889-6916-443f-9d83-dccd019d6544 nodeName:}" failed. No retries permitted until 2026-07-08 16:35:44.013077042 +0000 UTC m=+80.726438464 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/3898e889-6916-443f-9d83-dccd019d6544-node-exporter-tls") pod "node-exporter-zsdvl" (UID: "3898e889-6916-443f-9d83-dccd019d6544") : secret "node-exporter-tls" not found
Jul 08 16:35:43.514121 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.513643    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3898e889-6916-443f-9d83-dccd019d6544-metrics-client-ca\") pod \"node-exporter-zsdvl\" (UID: \"3898e889-6916-443f-9d83-dccd019d6544\") " pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:43.514121 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.513971    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/53b2bf60-5387-46eb-b264-e0410b55f8b9-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-b8mjr\" (UID: \"53b2bf60-5387-46eb-b264-e0410b55f8b9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b8mjr"
Jul 08 16:35:43.519496 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.519282    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/53b2bf60-5387-46eb-b264-e0410b55f8b9-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-b8mjr\" (UID: \"53b2bf60-5387-46eb-b264-e0410b55f8b9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b8mjr"
Jul 08 16:35:43.522823 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.522453    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3898e889-6916-443f-9d83-dccd019d6544-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zsdvl\" (UID: \"3898e889-6916-443f-9d83-dccd019d6544\") " pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:43.540271 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.540243    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8sfw\" (UniqueName: \"kubernetes.io/projected/53b2bf60-5387-46eb-b264-e0410b55f8b9-kube-api-access-m8sfw\") pod \"kube-state-metrics-7c9dbb8b7d-b8mjr\" (UID: \"53b2bf60-5387-46eb-b264-e0410b55f8b9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b8mjr"
Jul 08 16:35:43.551991 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.551966    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p28j\" (UniqueName: \"kubernetes.io/projected/3898e889-6916-443f-9d83-dccd019d6544-kube-api-access-4p28j\") pod \"node-exporter-zsdvl\" (UID: \"3898e889-6916-443f-9d83-dccd019d6544\") " pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:43.593019 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.592989    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-559cc97d7b-z4pnc"]
Jul 08 16:35:43.594489 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:35:43.594451    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf53fddb_4cb2_42c7_a274_6b74101a0f0e.slice/crio-61555c9d94a22bdf0282f0024ead84930865cc8da0954add96f24b8c565f276e WatchSource:0}: Error finding container 61555c9d94a22bdf0282f0024ead84930865cc8da0954add96f24b8c565f276e: Status 404 returned error can't find the container with id 61555c9d94a22bdf0282f0024ead84930865cc8da0954add96f24b8c565f276e
Jul 08 16:35:43.763143 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.763085    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:43.864003 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.863911    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:43.976815 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:43.976762    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:44.010474 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.010437    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/53b2bf60-5387-46eb-b264-e0410b55f8b9-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-b8mjr\" (UID: \"53b2bf60-5387-46eb-b264-e0410b55f8b9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b8mjr"
Jul 08 16:35:44.012866 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.012845    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/53b2bf60-5387-46eb-b264-e0410b55f8b9-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-b8mjr\" (UID: \"53b2bf60-5387-46eb-b264-e0410b55f8b9\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b8mjr"
Jul 08 16:35:44.066738 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.066698    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:44.111601 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.111573    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3898e889-6916-443f-9d83-dccd019d6544-node-exporter-tls\") pod \"node-exporter-zsdvl\" (UID: \"3898e889-6916-443f-9d83-dccd019d6544\") " pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:44.113870 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.113829    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3898e889-6916-443f-9d83-dccd019d6544-node-exporter-tls\") pod \"node-exporter-zsdvl\" (UID: \"3898e889-6916-443f-9d83-dccd019d6544\") " pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:44.237836 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.237792    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zsdvl"
Jul 08 16:35:44.245872 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.245845    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b8mjr"
Jul 08 16:35:44.250113 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:35:44.250082    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3898e889_6916_443f_9d83_dccd019d6544.slice/crio-b5e2a5dc243f5d854dad02d2babb45ed497bc468006573c81a04f8fa69a782f9 WatchSource:0}: Error finding container b5e2a5dc243f5d854dad02d2babb45ed497bc468006573c81a04f8fa69a782f9: Status 404 returned error can't find the container with id b5e2a5dc243f5d854dad02d2babb45ed497bc468006573c81a04f8fa69a782f9
Jul 08 16:35:44.327519 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.327475    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-559cc97d7b-z4pnc" event={"ID":"af53fddb-4cb2-42c7-a274-6b74101a0f0e","Type":"ContainerStarted","Data":"4c773b3a07305b54319ba58978e4b3745a132b82e61901a1185ca7a3abb57c46"}
Jul 08 16:35:44.327703 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.327547    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-559cc97d7b-z4pnc" event={"ID":"af53fddb-4cb2-42c7-a274-6b74101a0f0e","Type":"ContainerStarted","Data":"61555c9d94a22bdf0282f0024ead84930865cc8da0954add96f24b8c565f276e"}
Jul 08 16:35:44.329702 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.329653    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-24ccq" event={"ID":"59fa9d8c-d7a1-430d-bd58-560305b06ec9","Type":"ContainerStarted","Data":"3b334f77f5cfa75b011c840bedd69acc85581605d2db5721793571cee45d3ed0"}
Jul 08 16:35:44.331580 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.331522    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zsdvl" event={"ID":"3898e889-6916-443f-9d83-dccd019d6544","Type":"ContainerStarted","Data":"b5e2a5dc243f5d854dad02d2babb45ed497bc468006573c81a04f8fa69a782f9"}
Jul 08 16:35:44.334013 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.333974    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:44.349068 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.349013    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-559cc97d7b-z4pnc" podStartSLOduration=3.348990418 podStartE2EDuration="3.348990418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:44.34804544 +0000 UTC m=+81.061406869" watchObservedRunningTime="2026-07-08 16:35:44.348990418 +0000 UTC m=+81.062351847"
Jul 08 16:35:44.397864 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.397834    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b8mjr"]
Jul 08 16:35:44.408245 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:35:44.408215    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53b2bf60_5387_46eb_b264_e0410b55f8b9.slice/crio-a6d533b92fe2b85977e438a81db6de87e028aa4f5db58975ef6212bd2bd826cb WatchSource:0}: Error finding container a6d533b92fe2b85977e438a81db6de87e028aa4f5db58975ef6212bd2bd826cb: Status 404 returned error can't find the container with id a6d533b92fe2b85977e438a81db6de87e028aa4f5db58975ef6212bd2bd826cb
Jul 08 16:35:44.454367 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.454335    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 08 16:35:44.455668 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.455626    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:44.458211 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.458197    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.461258 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.460929    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 08 16:35:44.461258 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.460954    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 08 16:35:44.461258 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.461222    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-4wlpp\""
Jul 08 16:35:44.461422 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.461262    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 08 16:35:44.461479 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.461469    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 08 16:35:44.461559 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.461539    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 08 16:35:44.461615 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.461559    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 08 16:35:44.461615 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.461595    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 08 16:35:44.461756 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.461742    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 08 16:35:44.461822 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.461806    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 08 16:35:44.474189 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.474163    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 08 16:35:44.554299 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.554261    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:44.616534 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.616502    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.616534 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.616534    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.616745 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.616561    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.616745 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.616634    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.616745 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.616716    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-config-volume\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.616834 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.616747    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-config-out\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.616834 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.616778    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.616834 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.616805    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.616834 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.616825    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.616952 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.616844    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.616952 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.616868    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.616952 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.616884    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d428j\" (UniqueName: \"kubernetes.io/projected/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-kube-api-access-d428j\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.616952 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.616900    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-web-config\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.651802 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.651765    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:44.717870 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.717834    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-config-volume\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.717870 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.717873    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-config-out\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.718108 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.717897    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.718108 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.717916    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.718108 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.717934    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.718108 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.717953    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.718108 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.718092    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.718352 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.718143    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d428j\" (UniqueName: \"kubernetes.io/projected/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-kube-api-access-d428j\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.718352 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.718175    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-web-config\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.718352 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.718259    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.718352 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.718288    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.718352 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.718325    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.718352 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.718342    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.718352 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.718352    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.719062 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.719030    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.719709 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.719644    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.721138 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.721106    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.721337 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.721312    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.721407 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.721325    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-config-volume\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.721468 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.721410    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-web-config\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.721619 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.721600    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.721836 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.721815    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.722003 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.721981    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.722644 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.722627    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.722753 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.722727    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-config-out\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.728513 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.728493    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d428j\" (UniqueName: \"kubernetes.io/projected/35d5abaa-a8a0-46be-bc8e-7b789df4dfc6-kube-api-access-d428j\") pod \"alertmanager-main-0\" (UID: \"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.780629 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.780506    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 08 16:35:44.939562 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:44.939212    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 08 16:35:44.941873 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:35:44.941840    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35d5abaa_a8a0_46be_bc8e_7b789df4dfc6.slice/crio-d7ceedd77039284801af846dd094cc589ad6033aa08b8bae12b22aee6e957e61 WatchSource:0}: Error finding container d7ceedd77039284801af846dd094cc589ad6033aa08b8bae12b22aee6e957e61: Status 404 returned error can't find the container with id d7ceedd77039284801af846dd094cc589ad6033aa08b8bae12b22aee6e957e61
Jul 08 16:35:45.340704 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:45.340645    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-24ccq" event={"ID":"59fa9d8c-d7a1-430d-bd58-560305b06ec9","Type":"ContainerStarted","Data":"9d87f322dd8ecff4dd447ed5cdfdb79151538bfed1b2ed83724ebfa4d5aceb26"}
Jul 08 16:35:45.342393 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:45.342363    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b8mjr" event={"ID":"53b2bf60-5387-46eb-b264-e0410b55f8b9","Type":"ContainerStarted","Data":"a6d533b92fe2b85977e438a81db6de87e028aa4f5db58975ef6212bd2bd826cb"}
Jul 08 16:35:45.343905 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:45.343876    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6","Type":"ContainerStarted","Data":"d7ceedd77039284801af846dd094cc589ad6033aa08b8bae12b22aee6e957e61"}
Jul 08 16:35:46.350404 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:46.350368    2565 generic.go:356] "Generic (PLEG): container finished" podID="3898e889-6916-443f-9d83-dccd019d6544" containerID="e28a5e16919481e65d3124d69240d7caa49c65ced40353564dd9a99f728f23a7" exitCode=0
Jul 08 16:35:46.350890 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:46.350445    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zsdvl" event={"ID":"3898e889-6916-443f-9d83-dccd019d6544","Type":"ContainerDied","Data":"e28a5e16919481e65d3124d69240d7caa49c65ced40353564dd9a99f728f23a7"}
Jul 08 16:35:46.370237 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:46.370202    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:46.447538 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:46.447280    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:46.732585 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:46.732546    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:47.135850 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:47.135775    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:47.231366 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:47.231311    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:47.900537 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:47.900419    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:48.002641 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.002609    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:48.097724 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.097685    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-t9vpw"]
Jul 08 16:35:48.099565 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.099539    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:48.101128 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.101106    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-t9vpw"
Jul 08 16:35:48.103551 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.103532    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 08 16:35:48.103783 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.103767    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-7ndm4\""
Jul 08 16:35:48.111144 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.111123    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-t9vpw"]
Jul 08 16:35:48.223358 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.223324    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:48.255116 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.255074    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7blb\" (UniqueName: \"kubernetes.io/projected/b6393836-702c-4035-9e28-463bf38007f6-kube-api-access-p7blb\") pod \"monitoring-plugin-d5dcff7cf-t9vpw\" (UID: \"b6393836-702c-4035-9e28-463bf38007f6\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-t9vpw"
Jul 08 16:35:48.255292 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.255184    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6393836-702c-4035-9e28-463bf38007f6-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-t9vpw\" (UID: \"b6393836-702c-4035-9e28-463bf38007f6\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-t9vpw"
Jul 08 16:35:48.323665 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.323636    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:48.355862 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.355830    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6393836-702c-4035-9e28-463bf38007f6-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-t9vpw\" (UID: \"b6393836-702c-4035-9e28-463bf38007f6\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-t9vpw"
Jul 08 16:35:48.356017 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.355873    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7blb\" (UniqueName: \"kubernetes.io/projected/b6393836-702c-4035-9e28-463bf38007f6-kube-api-access-p7blb\") pod \"monitoring-plugin-d5dcff7cf-t9vpw\" (UID: \"b6393836-702c-4035-9e28-463bf38007f6\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-t9vpw"
Jul 08 16:35:48.358421 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.358399    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b6393836-702c-4035-9e28-463bf38007f6-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-t9vpw\" (UID: \"b6393836-702c-4035-9e28-463bf38007f6\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-t9vpw"
Jul 08 16:35:48.359569 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.359544    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-24ccq" event={"ID":"59fa9d8c-d7a1-430d-bd58-560305b06ec9","Type":"ContainerStarted","Data":"98a7aeef0255efa9e2adcb86d9299343e7845b0e1f58d71bfea3d61cbf97bdde"}
Jul 08 16:35:48.361628 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.361580    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zsdvl" event={"ID":"3898e889-6916-443f-9d83-dccd019d6544","Type":"ContainerStarted","Data":"e453018c60c31b7256a70515669da7456bd588470087c32c6c575c3917e9a91a"}
Jul 08 16:35:48.361749 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.361630    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zsdvl" event={"ID":"3898e889-6916-443f-9d83-dccd019d6544","Type":"ContainerStarted","Data":"ae85e6e10346e28ae3c8abbd995aa984c60fc9b5641c4d1fae7a76a73831cee7"}
Jul 08 16:35:48.363729 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.363700    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b8mjr" event={"ID":"53b2bf60-5387-46eb-b264-e0410b55f8b9","Type":"ContainerStarted","Data":"8eca4bc8993e959287e7adad9225405c35707b8f784a24d793a9db0c176b6520"}
Jul 08 16:35:48.363817 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.363756    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b8mjr" event={"ID":"53b2bf60-5387-46eb-b264-e0410b55f8b9","Type":"ContainerStarted","Data":"d0989b17ac8257b5aa0d3924fcafed42c2b86ca396a14beee81f2411e23b21f2"}
Jul 08 16:35:48.363817 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.363787    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b8mjr" event={"ID":"53b2bf60-5387-46eb-b264-e0410b55f8b9","Type":"ContainerStarted","Data":"6c8a307934051d2623497a2540c7e11309c2aa9df2f91829bbc0e2f8d531695d"}
Jul 08 16:35:48.365119 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.365097    2565 generic.go:356] "Generic (PLEG): container finished" podID="35d5abaa-a8a0-46be-bc8e-7b789df4dfc6" containerID="797245d6fb3bc5143936b3da92be6d1da6eeeff61288be14fff7b1312ced233b" exitCode=0
Jul 08 16:35:48.365187 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.365165    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6","Type":"ContainerDied","Data":"797245d6fb3bc5143936b3da92be6d1da6eeeff61288be14fff7b1312ced233b"}
Jul 08 16:35:48.367296 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.367278    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7blb\" (UniqueName: \"kubernetes.io/projected/b6393836-702c-4035-9e28-463bf38007f6-kube-api-access-p7blb\") pod \"monitoring-plugin-d5dcff7cf-t9vpw\" (UID: \"b6393836-702c-4035-9e28-463bf38007f6\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-t9vpw"
Jul 08 16:35:48.384403 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.384359    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-24ccq" podStartSLOduration=37.39849547 podStartE2EDuration="45.384345358s" totalImagesPullingTime="7.985849888s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:35:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:48.383327609 +0000 UTC m=+85.096689039" watchObservedRunningTime="2026-07-08 16:35:48.384345358 +0000 UTC m=+85.097706784"
Jul 08 16:35:48.411014 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.410975    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-t9vpw"
Jul 08 16:35:48.412227 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.412180    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zsdvl" podStartSLOduration=4.26982152 podStartE2EDuration="5.412164115s" totalImagesPullingTime="1.065699743s" totalInitContainerRuntime="76.642852ms" isStatefulPod=true podCreationTimestamp="2026-07-08 16:35:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:48.4113067 +0000 UTC m=+85.124668152" watchObservedRunningTime="2026-07-08 16:35:48.412164115 +0000 UTC m=+85.125525542"
Jul 08 16:35:48.437044 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.436994    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-b8mjr" podStartSLOduration=2.172698354 podStartE2EDuration="5.4369775s" totalImagesPullingTime="3.264279146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:48.435310752 +0000 UTC m=+85.148672181" watchObservedRunningTime="2026-07-08 16:35:48.4369775 +0000 UTC m=+85.150338926"
Jul 08 16:35:48.545408 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.545386    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-t9vpw"]
Jul 08 16:35:48.547357 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:35:48.547324    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6393836_702c_4035_9e28_463bf38007f6.slice/crio-ce7f0c32677f8ce0ccc5d69ac98b0ba7973aab4aff5d84719acb9fa27507c273 WatchSource:0}: Error finding container ce7f0c32677f8ce0ccc5d69ac98b0ba7973aab4aff5d84719acb9fa27507c273: Status 404 returned error can't find the container with id ce7f0c32677f8ce0ccc5d69ac98b0ba7973aab4aff5d84719acb9fa27507c273
Jul 08 16:35:48.765606 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.765524    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:48.860160 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:48.860126    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:49.370276 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:49.370180    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-t9vpw" event={"ID":"b6393836-702c-4035-9e28-463bf38007f6","Type":"ContainerStarted","Data":"ce7f0c32677f8ce0ccc5d69ac98b0ba7973aab4aff5d84719acb9fa27507c273"}
Jul 08 16:35:49.791575 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:49.791539    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:49.900892 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:49.900853    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:50.403759 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:50.403726    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-559cc97d7b-z4pnc"]
Jul 08 16:35:51.379742 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:51.379631    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-t9vpw" event={"ID":"b6393836-702c-4035-9e28-463bf38007f6","Type":"ContainerStarted","Data":"2a811448905193adcb7c28ed926b386cbb232052a2a802e14deadddd0177b7a7"}
Jul 08 16:35:51.379910 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:51.379811    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-t9vpw"
Jul 08 16:35:51.383067 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:51.383044    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6","Type":"ContainerStarted","Data":"3ef58d85a65c0e7e12f13afada14519584bdae8084310a40c34c0c041a76658c"}
Jul 08 16:35:51.383215 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:51.383098    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6","Type":"ContainerStarted","Data":"008fd8ae72a7a76c7bda704d555b47e74f1e824d8ecc1c3b20e26091183804f0"}
Jul 08 16:35:51.383215 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:51.383134    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6","Type":"ContainerStarted","Data":"294b67b7c80af0eed5ac266d785afff70850f64bf1e5d2183311fb3690d5232b"}
Jul 08 16:35:51.383215 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:51.383160    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6","Type":"ContainerStarted","Data":"0e500fe31c204c7ca685cfaa1d1bc477dcdc58abcbbd56e072901490f2b918ac"}
Jul 08 16:35:51.383215 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:51.383183    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6","Type":"ContainerStarted","Data":"9f669cd9ba56cc0c226910306d414abd3659d6fef64fd3810ebb07930022209b"}
Jul 08 16:35:51.385521 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:51.385502    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-t9vpw"
Jul 08 16:35:51.406755 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:51.406709    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-t9vpw" podStartSLOduration=1.161324477 podStartE2EDuration="3.406667203s" totalImagesPullingTime="2.245342726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:51.40620587 +0000 UTC m=+88.119567318" watchObservedRunningTime="2026-07-08 16:35:51.406667203 +0000 UTC m=+88.120028630"
Jul 08 16:35:51.860169 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:51.860140    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-559cc97d7b-z4pnc"
Jul 08 16:35:52.135325 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:52.135237    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:52.426330 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:52.426284    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:52.599452 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:52.599409    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:53.059692 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:53.059647    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:53.161189 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:53.161153    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:35:53.394614 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:53.394528    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35d5abaa-a8a0-46be-bc8e-7b789df4dfc6","Type":"ContainerStarted","Data":"72fd2d28a527e37ac15f3196b7509301fbf12ddbbd6707d61aecf2c146b8b2d5"}
Jul 08 16:35:53.432497 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:53.432435    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.919051525 podStartE2EDuration="9.432367789s" totalImagesPullingTime="6.489286693s" totalInitContainerRuntime="24.029571ms" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:53.425472293 +0000 UTC m=+90.138833740" watchObservedRunningTime="2026-07-08 16:35:53.432367789 +0000 UTC m=+90.145729216"
Jul 08 16:35:55.613454 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:55.613394    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-795cc6499b-97mrt" podUID="0483b06a-6fc8-4e43-a1f8-99a63e9e61cb" containerName="registry" containerID="cri-o://88410f344f16839a25e8d5e5352227d7c9759ee0dd16d43a87e557bf885d582d" gracePeriod=30
Jul 08 16:35:55.864061 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:55.864001    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:35:56.025926 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.025888    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-ca-trust-extracted\") pod \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") "
Jul 08 16:35:56.026117 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.025975    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-image-registry-private-configuration\") pod \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") "
Jul 08 16:35:56.026117 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.026000    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-registry-certificates\") pod \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") "
Jul 08 16:35:56.026117 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.026039    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4fnw\" (UniqueName: \"kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-kube-api-access-p4fnw\") pod \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") "
Jul 08 16:35:56.026117 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.026061    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-trusted-ca\") pod \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") "
Jul 08 16:35:56.026117 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.026085    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-installation-pull-secrets\") pod \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") "
Jul 08 16:35:56.026362 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.026118    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-bound-sa-token\") pod \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") "
Jul 08 16:35:56.026362 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.026141    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-registry-tls\") pod \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\" (UID: \"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb\") "
Jul 08 16:35:56.026490 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.026461    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "0483b06a-6fc8-4e43-a1f8-99a63e9e61cb" (UID: "0483b06a-6fc8-4e43-a1f8-99a63e9e61cb"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:35:56.026792 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.026762    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "0483b06a-6fc8-4e43-a1f8-99a63e9e61cb" (UID: "0483b06a-6fc8-4e43-a1f8-99a63e9e61cb"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:35:56.028512 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.028480    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "0483b06a-6fc8-4e43-a1f8-99a63e9e61cb" (UID: "0483b06a-6fc8-4e43-a1f8-99a63e9e61cb"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:35:56.028634 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.028491    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "0483b06a-6fc8-4e43-a1f8-99a63e9e61cb" (UID: "0483b06a-6fc8-4e43-a1f8-99a63e9e61cb"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:35:56.028725 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.028661    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-kube-api-access-p4fnw" (OuterVolumeSpecName: "kube-api-access-p4fnw") pod "0483b06a-6fc8-4e43-a1f8-99a63e9e61cb" (UID: "0483b06a-6fc8-4e43-a1f8-99a63e9e61cb"). InnerVolumeSpecName "kube-api-access-p4fnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:35:56.028786 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.028761    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0483b06a-6fc8-4e43-a1f8-99a63e9e61cb" (UID: "0483b06a-6fc8-4e43-a1f8-99a63e9e61cb"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:35:56.028908 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.028890    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "0483b06a-6fc8-4e43-a1f8-99a63e9e61cb" (UID: "0483b06a-6fc8-4e43-a1f8-99a63e9e61cb"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:35:56.034502 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.034474    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "0483b06a-6fc8-4e43-a1f8-99a63e9e61cb" (UID: "0483b06a-6fc8-4e43-a1f8-99a63e9e61cb"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:35:56.127213 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.127146    2565 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-image-registry-private-configuration\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:35:56.127213 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.127168    2565 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-registry-certificates\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:35:56.127213 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.127178    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4fnw\" (UniqueName: \"kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-kube-api-access-p4fnw\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:35:56.127213 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.127189    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-trusted-ca\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:35:56.127213 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.127197    2565 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-installation-pull-secrets\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:35:56.127213 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.127205    2565 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-bound-sa-token\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:35:56.127213 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.127213    2565 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-registry-tls\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:35:56.127497 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.127221    2565 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb-ca-trust-extracted\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:35:56.410254 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.410174    2565 generic.go:356] "Generic (PLEG): container finished" podID="0483b06a-6fc8-4e43-a1f8-99a63e9e61cb" containerID="88410f344f16839a25e8d5e5352227d7c9759ee0dd16d43a87e557bf885d582d" exitCode=0
Jul 08 16:35:56.410254 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.410233    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-795cc6499b-97mrt" event={"ID":"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb","Type":"ContainerDied","Data":"88410f344f16839a25e8d5e5352227d7c9759ee0dd16d43a87e557bf885d582d"}
Jul 08 16:35:56.410254 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.410235    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-795cc6499b-97mrt"
Jul 08 16:35:56.410463 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.410292    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-795cc6499b-97mrt" event={"ID":"0483b06a-6fc8-4e43-a1f8-99a63e9e61cb","Type":"ContainerDied","Data":"51f255a50ac4c5e1b2eb371b3bff999e50b27de698508b78c4d583bc28a0603b"}
Jul 08 16:35:56.410463 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.410313    2565 scope.go:117] "RemoveContainer" containerID="88410f344f16839a25e8d5e5352227d7c9759ee0dd16d43a87e557bf885d582d"
Jul 08 16:35:56.419221 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.419202    2565 scope.go:117] "RemoveContainer" containerID="88410f344f16839a25e8d5e5352227d7c9759ee0dd16d43a87e557bf885d582d"
Jul 08 16:35:56.419476 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:35:56.419449    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88410f344f16839a25e8d5e5352227d7c9759ee0dd16d43a87e557bf885d582d\": container with ID starting with 88410f344f16839a25e8d5e5352227d7c9759ee0dd16d43a87e557bf885d582d not found: ID does not exist" containerID="88410f344f16839a25e8d5e5352227d7c9759ee0dd16d43a87e557bf885d582d"
Jul 08 16:35:56.419521 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.419475    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88410f344f16839a25e8d5e5352227d7c9759ee0dd16d43a87e557bf885d582d"} err="failed to get container status \"88410f344f16839a25e8d5e5352227d7c9759ee0dd16d43a87e557bf885d582d\": rpc error: code = NotFound desc = could not find container \"88410f344f16839a25e8d5e5352227d7c9759ee0dd16d43a87e557bf885d582d\": container with ID starting with 88410f344f16839a25e8d5e5352227d7c9759ee0dd16d43a87e557bf885d582d not found: ID does not exist"
Jul 08 16:35:56.441211 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.441186    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-795cc6499b-97mrt"]
Jul 08 16:35:56.448531 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:56.448511    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-795cc6499b-97mrt"]
Jul 08 16:35:57.820415 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:35:57.820378    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0483b06a-6fc8-4e43-a1f8-99a63e9e61cb" path="/var/lib/kubelet/pods/0483b06a-6fc8-4e43-a1f8-99a63e9e61cb/volumes"
Jul 08 16:36:00.150539 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:00.150507    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:00.251524 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:00.251474    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:00.935809 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:00.935775    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59d4bb4df4-qjlqg"]
Jul 08 16:36:01.375365 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:01.375276    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:01.437974 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:01.437944    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:07.157985 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:07.157948    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:07.272422 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:07.272388    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:12.315759 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:12.315730    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8m52j"
Jul 08 16:36:15.425901 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:15.425858    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-559cc97d7b-z4pnc" podUID="af53fddb-4cb2-42c7-a274-6b74101a0f0e" containerName="console" containerID="cri-o://4c773b3a07305b54319ba58978e4b3745a132b82e61901a1185ca7a3abb57c46" gracePeriod=15
Jul 08 16:36:15.728879 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:15.728849    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-559cc97d7b-z4pnc"
Jul 08 16:36:15.778394 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:15.778362    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af53fddb-4cb2-42c7-a274-6b74101a0f0e-oauth-serving-cert\") pod \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\" (UID: \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\") "
Jul 08 16:36:15.778394 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:15.778400    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hl9ds\" (UniqueName: \"kubernetes.io/projected/af53fddb-4cb2-42c7-a274-6b74101a0f0e-kube-api-access-hl9ds\") pod \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\" (UID: \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\") "
Jul 08 16:36:15.778626 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:15.778432    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af53fddb-4cb2-42c7-a274-6b74101a0f0e-trusted-ca-bundle\") pod \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\" (UID: \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\") "
Jul 08 16:36:15.778626 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:15.778452    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af53fddb-4cb2-42c7-a274-6b74101a0f0e-service-ca\") pod \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\" (UID: \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\") "
Jul 08 16:36:15.778626 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:15.778484    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af53fddb-4cb2-42c7-a274-6b74101a0f0e-console-oauth-config\") pod \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\" (UID: \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\") "
Jul 08 16:36:15.778626 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:15.778505    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af53fddb-4cb2-42c7-a274-6b74101a0f0e-console-config\") pod \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\" (UID: \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\") "
Jul 08 16:36:15.778626 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:15.778546    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af53fddb-4cb2-42c7-a274-6b74101a0f0e-console-serving-cert\") pod \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\" (UID: \"af53fddb-4cb2-42c7-a274-6b74101a0f0e\") "
Jul 08 16:36:15.778908 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:15.778795    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af53fddb-4cb2-42c7-a274-6b74101a0f0e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "af53fddb-4cb2-42c7-a274-6b74101a0f0e" (UID: "af53fddb-4cb2-42c7-a274-6b74101a0f0e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:36:15.778908 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:15.778884    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af53fddb-4cb2-42c7-a274-6b74101a0f0e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "af53fddb-4cb2-42c7-a274-6b74101a0f0e" (UID: "af53fddb-4cb2-42c7-a274-6b74101a0f0e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:36:15.778908 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:15.778896    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af53fddb-4cb2-42c7-a274-6b74101a0f0e-service-ca" (OuterVolumeSpecName: "service-ca") pod "af53fddb-4cb2-42c7-a274-6b74101a0f0e" (UID: "af53fddb-4cb2-42c7-a274-6b74101a0f0e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:36:15.779109 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:15.779086    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af53fddb-4cb2-42c7-a274-6b74101a0f0e-console-config" (OuterVolumeSpecName: "console-config") pod "af53fddb-4cb2-42c7-a274-6b74101a0f0e" (UID: "af53fddb-4cb2-42c7-a274-6b74101a0f0e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:36:15.780770 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:15.780744    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af53fddb-4cb2-42c7-a274-6b74101a0f0e-kube-api-access-hl9ds" (OuterVolumeSpecName: "kube-api-access-hl9ds") pod "af53fddb-4cb2-42c7-a274-6b74101a0f0e" (UID: "af53fddb-4cb2-42c7-a274-6b74101a0f0e"). InnerVolumeSpecName "kube-api-access-hl9ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:36:15.780770 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:15.780751    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af53fddb-4cb2-42c7-a274-6b74101a0f0e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "af53fddb-4cb2-42c7-a274-6b74101a0f0e" (UID: "af53fddb-4cb2-42c7-a274-6b74101a0f0e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:36:15.780891 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:15.780772    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af53fddb-4cb2-42c7-a274-6b74101a0f0e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "af53fddb-4cb2-42c7-a274-6b74101a0f0e" (UID: "af53fddb-4cb2-42c7-a274-6b74101a0f0e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:36:15.880068 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:15.880033    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af53fddb-4cb2-42c7-a274-6b74101a0f0e-trusted-ca-bundle\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:36:15.880068 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:15.880062    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/af53fddb-4cb2-42c7-a274-6b74101a0f0e-service-ca\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:36:15.880068 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:15.880073    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/af53fddb-4cb2-42c7-a274-6b74101a0f0e-console-oauth-config\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:36:15.880283 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:15.880082    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/af53fddb-4cb2-42c7-a274-6b74101a0f0e-console-config\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:36:15.880283 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:15.880092    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/af53fddb-4cb2-42c7-a274-6b74101a0f0e-console-serving-cert\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:36:15.880283 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:15.880122    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/af53fddb-4cb2-42c7-a274-6b74101a0f0e-oauth-serving-cert\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:36:15.880283 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:15.880132    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl9ds\" (UniqueName: \"kubernetes.io/projected/af53fddb-4cb2-42c7-a274-6b74101a0f0e-kube-api-access-hl9ds\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:36:16.491769 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:16.491735    2565 generic.go:356] "Generic (PLEG): container finished" podID="af53fddb-4cb2-42c7-a274-6b74101a0f0e" containerID="4c773b3a07305b54319ba58978e4b3745a132b82e61901a1185ca7a3abb57c46" exitCode=0
Jul 08 16:36:16.492236 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:16.491785    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-559cc97d7b-z4pnc"
Jul 08 16:36:16.492236 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:16.491799    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-559cc97d7b-z4pnc" event={"ID":"af53fddb-4cb2-42c7-a274-6b74101a0f0e","Type":"ContainerDied","Data":"4c773b3a07305b54319ba58978e4b3745a132b82e61901a1185ca7a3abb57c46"}
Jul 08 16:36:16.492236 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:16.491849    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-559cc97d7b-z4pnc" event={"ID":"af53fddb-4cb2-42c7-a274-6b74101a0f0e","Type":"ContainerDied","Data":"61555c9d94a22bdf0282f0024ead84930865cc8da0954add96f24b8c565f276e"}
Jul 08 16:36:16.492236 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:16.491877    2565 scope.go:117] "RemoveContainer" containerID="4c773b3a07305b54319ba58978e4b3745a132b82e61901a1185ca7a3abb57c46"
Jul 08 16:36:16.500350 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:16.500328    2565 scope.go:117] "RemoveContainer" containerID="4c773b3a07305b54319ba58978e4b3745a132b82e61901a1185ca7a3abb57c46"
Jul 08 16:36:16.500604 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:36:16.500588    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c773b3a07305b54319ba58978e4b3745a132b82e61901a1185ca7a3abb57c46\": container with ID starting with 4c773b3a07305b54319ba58978e4b3745a132b82e61901a1185ca7a3abb57c46 not found: ID does not exist" containerID="4c773b3a07305b54319ba58978e4b3745a132b82e61901a1185ca7a3abb57c46"
Jul 08 16:36:16.500659 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:16.500612    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c773b3a07305b54319ba58978e4b3745a132b82e61901a1185ca7a3abb57c46"} err="failed to get container status \"4c773b3a07305b54319ba58978e4b3745a132b82e61901a1185ca7a3abb57c46\": rpc error: code = NotFound desc = could not find container \"4c773b3a07305b54319ba58978e4b3745a132b82e61901a1185ca7a3abb57c46\": container with ID starting with 4c773b3a07305b54319ba58978e4b3745a132b82e61901a1185ca7a3abb57c46 not found: ID does not exist"
Jul 08 16:36:16.510397 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:16.510378    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-559cc97d7b-z4pnc"]
Jul 08 16:36:16.515366 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:16.515347    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-559cc97d7b-z4pnc"]
Jul 08 16:36:17.739525 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:17.739483    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:17.820549 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:17.820508    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af53fddb-4cb2-42c7-a274-6b74101a0f0e" path="/var/lib/kubelet/pods/af53fddb-4cb2-42c7-a274-6b74101a0f0e/volumes"
Jul 08 16:36:17.839919 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:17.839889    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:17.910966 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:17.910927    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:23.140160 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:23.140122    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:23.241823 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:23.241790    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:25.955834 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:25.955786    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-59d4bb4df4-qjlqg" podUID="9fbd600c-a512-46e6-af5f-c09771373fe9" containerName="console" containerID="cri-o://827789282b2907f15bb5e6e62f2c134dc1321660ea284aef7fe1fc695f80ac7f" gracePeriod=15
Jul 08 16:36:26.194050 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:26.194020    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d4bb4df4-qjlqg"
Jul 08 16:36:26.260263 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:26.260179    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96mt6\" (UniqueName: \"kubernetes.io/projected/9fbd600c-a512-46e6-af5f-c09771373fe9-kube-api-access-96mt6\") pod \"9fbd600c-a512-46e6-af5f-c09771373fe9\" (UID: \"9fbd600c-a512-46e6-af5f-c09771373fe9\") "
Jul 08 16:36:26.260263 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:26.260217    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fbd600c-a512-46e6-af5f-c09771373fe9-service-ca\") pod \"9fbd600c-a512-46e6-af5f-c09771373fe9\" (UID: \"9fbd600c-a512-46e6-af5f-c09771373fe9\") "
Jul 08 16:36:26.260263 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:26.260251    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fbd600c-a512-46e6-af5f-c09771373fe9-console-oauth-config\") pod \"9fbd600c-a512-46e6-af5f-c09771373fe9\" (UID: \"9fbd600c-a512-46e6-af5f-c09771373fe9\") "
Jul 08 16:36:26.260520 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:26.260298    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fbd600c-a512-46e6-af5f-c09771373fe9-console-config\") pod \"9fbd600c-a512-46e6-af5f-c09771373fe9\" (UID: \"9fbd600c-a512-46e6-af5f-c09771373fe9\") "
Jul 08 16:36:26.260520 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:26.260320    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fbd600c-a512-46e6-af5f-c09771373fe9-console-serving-cert\") pod \"9fbd600c-a512-46e6-af5f-c09771373fe9\" (UID: \"9fbd600c-a512-46e6-af5f-c09771373fe9\") "
Jul 08 16:36:26.260520 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:26.260339    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fbd600c-a512-46e6-af5f-c09771373fe9-oauth-serving-cert\") pod \"9fbd600c-a512-46e6-af5f-c09771373fe9\" (UID: \"9fbd600c-a512-46e6-af5f-c09771373fe9\") "
Jul 08 16:36:26.260758 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:26.260729    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbd600c-a512-46e6-af5f-c09771373fe9-console-config" (OuterVolumeSpecName: "console-config") pod "9fbd600c-a512-46e6-af5f-c09771373fe9" (UID: "9fbd600c-a512-46e6-af5f-c09771373fe9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:36:26.260873 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:26.260807    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbd600c-a512-46e6-af5f-c09771373fe9-service-ca" (OuterVolumeSpecName: "service-ca") pod "9fbd600c-a512-46e6-af5f-c09771373fe9" (UID: "9fbd600c-a512-46e6-af5f-c09771373fe9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:36:26.260873 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:26.260817    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fbd600c-a512-46e6-af5f-c09771373fe9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9fbd600c-a512-46e6-af5f-c09771373fe9" (UID: "9fbd600c-a512-46e6-af5f-c09771373fe9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:36:26.262524 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:26.262498    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fbd600c-a512-46e6-af5f-c09771373fe9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9fbd600c-a512-46e6-af5f-c09771373fe9" (UID: "9fbd600c-a512-46e6-af5f-c09771373fe9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:36:26.262597 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:26.262575    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fbd600c-a512-46e6-af5f-c09771373fe9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9fbd600c-a512-46e6-af5f-c09771373fe9" (UID: "9fbd600c-a512-46e6-af5f-c09771373fe9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:36:26.262639 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:26.262621    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fbd600c-a512-46e6-af5f-c09771373fe9-kube-api-access-96mt6" (OuterVolumeSpecName: "kube-api-access-96mt6") pod "9fbd600c-a512-46e6-af5f-c09771373fe9" (UID: "9fbd600c-a512-46e6-af5f-c09771373fe9"). InnerVolumeSpecName "kube-api-access-96mt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:36:26.361056 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:26.361024    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96mt6\" (UniqueName: \"kubernetes.io/projected/9fbd600c-a512-46e6-af5f-c09771373fe9-kube-api-access-96mt6\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:36:26.361056 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:26.361052    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fbd600c-a512-46e6-af5f-c09771373fe9-service-ca\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:36:26.361056 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:26.361064    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fbd600c-a512-46e6-af5f-c09771373fe9-console-oauth-config\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:36:26.361056 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:26.361073    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fbd600c-a512-46e6-af5f-c09771373fe9-console-config\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:36:26.361324 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:26.361081    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fbd600c-a512-46e6-af5f-c09771373fe9-console-serving-cert\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:36:26.361324 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:26.361090    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fbd600c-a512-46e6-af5f-c09771373fe9-oauth-serving-cert\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:36:26.527903 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:26.527820    2565 generic.go:356] "Generic (PLEG): container finished" podID="9fbd600c-a512-46e6-af5f-c09771373fe9" containerID="827789282b2907f15bb5e6e62f2c134dc1321660ea284aef7fe1fc695f80ac7f" exitCode=0
Jul 08 16:36:26.527903 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:26.527867    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d4bb4df4-qjlqg"
Jul 08 16:36:26.527903 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:26.527892    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d4bb4df4-qjlqg" event={"ID":"9fbd600c-a512-46e6-af5f-c09771373fe9","Type":"ContainerDied","Data":"827789282b2907f15bb5e6e62f2c134dc1321660ea284aef7fe1fc695f80ac7f"}
Jul 08 16:36:26.528126 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:26.527930    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d4bb4df4-qjlqg" event={"ID":"9fbd600c-a512-46e6-af5f-c09771373fe9","Type":"ContainerDied","Data":"81f6d61da220b8d4e7428507aaa6d9973589ffe60aefa7d7b8684189e4d88770"}
Jul 08 16:36:26.528126 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:26.527950    2565 scope.go:117] "RemoveContainer" containerID="827789282b2907f15bb5e6e62f2c134dc1321660ea284aef7fe1fc695f80ac7f"
Jul 08 16:36:26.536427 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:26.536334    2565 scope.go:117] "RemoveContainer" containerID="827789282b2907f15bb5e6e62f2c134dc1321660ea284aef7fe1fc695f80ac7f"
Jul 08 16:36:26.536649 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:36:26.536623    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"827789282b2907f15bb5e6e62f2c134dc1321660ea284aef7fe1fc695f80ac7f\": container with ID starting with 827789282b2907f15bb5e6e62f2c134dc1321660ea284aef7fe1fc695f80ac7f not found: ID does not exist" containerID="827789282b2907f15bb5e6e62f2c134dc1321660ea284aef7fe1fc695f80ac7f"
Jul 08 16:36:26.536764 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:26.536656    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"827789282b2907f15bb5e6e62f2c134dc1321660ea284aef7fe1fc695f80ac7f"} err="failed to get container status \"827789282b2907f15bb5e6e62f2c134dc1321660ea284aef7fe1fc695f80ac7f\": rpc error: code = NotFound desc = could not find container \"827789282b2907f15bb5e6e62f2c134dc1321660ea284aef7fe1fc695f80ac7f\": container with ID starting with 827789282b2907f15bb5e6e62f2c134dc1321660ea284aef7fe1fc695f80ac7f not found: ID does not exist"
Jul 08 16:36:26.552489 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:26.552463    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59d4bb4df4-qjlqg"]
Jul 08 16:36:26.555292 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:26.555263    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59d4bb4df4-qjlqg"]
Jul 08 16:36:27.820818 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:27.820785    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fbd600c-a512-46e6-af5f-c09771373fe9" path="/var/lib/kubelet/pods/9fbd600c-a512-46e6-af5f-c09771373fe9/volumes"
Jul 08 16:36:33.834467 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:33.834431    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:33.935312 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:33.935279    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:38.650520 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:38.650478    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:36:38.727663 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:36:38.727630    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:02.747865 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:02.747827    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:03.032376 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:03.032299    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:07.880696 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.880643    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-646c8d68d4-kd7vt"]
Jul 08 16:37:07.881146 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.880933    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af53fddb-4cb2-42c7-a274-6b74101a0f0e" containerName="console"
Jul 08 16:37:07.881146 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.880948    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="af53fddb-4cb2-42c7-a274-6b74101a0f0e" containerName="console"
Jul 08 16:37:07.881146 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.880960    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0483b06a-6fc8-4e43-a1f8-99a63e9e61cb" containerName="registry"
Jul 08 16:37:07.881146 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.880965    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0483b06a-6fc8-4e43-a1f8-99a63e9e61cb" containerName="registry"
Jul 08 16:37:07.881146 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.880982    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fbd600c-a512-46e6-af5f-c09771373fe9" containerName="console"
Jul 08 16:37:07.881146 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.880988    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fbd600c-a512-46e6-af5f-c09771373fe9" containerName="console"
Jul 08 16:37:07.881146 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.881033    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="af53fddb-4cb2-42c7-a274-6b74101a0f0e" containerName="console"
Jul 08 16:37:07.881146 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.881041    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0483b06a-6fc8-4e43-a1f8-99a63e9e61cb" containerName="registry"
Jul 08 16:37:07.881146 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.881048    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fbd600c-a512-46e6-af5f-c09771373fe9" containerName="console"
Jul 08 16:37:07.883987 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.883963    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt"
Jul 08 16:37:07.889155 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.889120    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 08 16:37:07.889339 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.889320    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-mfgt6\""
Jul 08 16:37:07.889409 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.889389    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee-secret-telemeter-client\") pod \"telemeter-client-646c8d68d4-kd7vt\" (UID: \"a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee\") " pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt"
Jul 08 16:37:07.889458 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.889425    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee-federate-client-tls\") pod \"telemeter-client-646c8d68d4-kd7vt\" (UID: \"a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee\") " pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt"
Jul 08 16:37:07.889458 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.889452    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee-telemeter-trusted-ca-bundle\") pod \"telemeter-client-646c8d68d4-kd7vt\" (UID: \"a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee\") " pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt"
Jul 08 16:37:07.889552 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.889480    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-646c8d68d4-kd7vt\" (UID: \"a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee\") " pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt"
Jul 08 16:37:07.889606 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.889573    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee-metrics-client-ca\") pod \"telemeter-client-646c8d68d4-kd7vt\" (UID: \"a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee\") " pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt"
Jul 08 16:37:07.889657 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.889618    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 08 16:37:07.889730 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.889690    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee-serving-certs-ca-bundle\") pod \"telemeter-client-646c8d68d4-kd7vt\" (UID: \"a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee\") " pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt"
Jul 08 16:37:07.889782 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.889762    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bxxq\" (UniqueName: \"kubernetes.io/projected/a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee-kube-api-access-7bxxq\") pod \"telemeter-client-646c8d68d4-kd7vt\" (UID: \"a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee\") " pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt"
Jul 08 16:37:07.889838 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.889816    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee-telemeter-client-tls\") pod \"telemeter-client-646c8d68d4-kd7vt\" (UID: \"a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee\") " pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt"
Jul 08 16:37:07.890078 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.890058    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 08 16:37:07.890504 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.890329    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 08 16:37:07.890795 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.890780    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 08 16:37:07.907731 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.907709    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 08 16:37:07.911044 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.911022    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-646c8d68d4-kd7vt"]
Jul 08 16:37:07.991131 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.991099    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee-secret-telemeter-client\") pod \"telemeter-client-646c8d68d4-kd7vt\" (UID: \"a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee\") " pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt"
Jul 08 16:37:07.991131 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.991130    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee-federate-client-tls\") pod \"telemeter-client-646c8d68d4-kd7vt\" (UID: \"a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee\") " pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt"
Jul 08 16:37:07.991332 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.991150    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee-telemeter-trusted-ca-bundle\") pod \"telemeter-client-646c8d68d4-kd7vt\" (UID: \"a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee\") " pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt"
Jul 08 16:37:07.991332 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.991183    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-646c8d68d4-kd7vt\" (UID: \"a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee\") " pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt"
Jul 08 16:37:07.991332 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.991216    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee-metrics-client-ca\") pod \"telemeter-client-646c8d68d4-kd7vt\" (UID: \"a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee\") " pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt"
Jul 08 16:37:07.991332 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.991244    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee-serving-certs-ca-bundle\") pod \"telemeter-client-646c8d68d4-kd7vt\" (UID: \"a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee\") " pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt"
Jul 08 16:37:07.991332 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.991284    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bxxq\" (UniqueName: \"kubernetes.io/projected/a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee-kube-api-access-7bxxq\") pod \"telemeter-client-646c8d68d4-kd7vt\" (UID: \"a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee\") " pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt"
Jul 08 16:37:07.991332 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.991330    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee-telemeter-client-tls\") pod \"telemeter-client-646c8d68d4-kd7vt\" (UID: \"a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee\") " pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt"
Jul 08 16:37:07.992082 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.992057    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee-metrics-client-ca\") pod \"telemeter-client-646c8d68d4-kd7vt\" (UID: \"a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee\") " pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt"
Jul 08 16:37:07.992257 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.992230    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee-serving-certs-ca-bundle\") pod \"telemeter-client-646c8d68d4-kd7vt\" (UID: \"a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee\") " pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt"
Jul 08 16:37:07.992334 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.992234    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee-telemeter-trusted-ca-bundle\") pod \"telemeter-client-646c8d68d4-kd7vt\" (UID: \"a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee\") " pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt"
Jul 08 16:37:07.993829 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.993805    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee-federate-client-tls\") pod \"telemeter-client-646c8d68d4-kd7vt\" (UID: \"a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee\") " pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt"
Jul 08 16:37:07.993985 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.993966    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-646c8d68d4-kd7vt\" (UID: \"a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee\") " pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt"
Jul 08 16:37:07.994042 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.993987    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee-secret-telemeter-client\") pod \"telemeter-client-646c8d68d4-kd7vt\" (UID: \"a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee\") " pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt"
Jul 08 16:37:07.994042 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:07.993998    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee-telemeter-client-tls\") pod \"telemeter-client-646c8d68d4-kd7vt\" (UID: \"a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee\") " pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt"
Jul 08 16:37:08.002044 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:08.002026    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bxxq\" (UniqueName: \"kubernetes.io/projected/a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee-kube-api-access-7bxxq\") pod \"telemeter-client-646c8d68d4-kd7vt\" (UID: \"a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee\") " pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt"
Jul 08 16:37:08.193423 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:08.193379    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt"
Jul 08 16:37:08.324197 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:08.324170    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-646c8d68d4-kd7vt"]
Jul 08 16:37:08.325721 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:37:08.325695    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1bc2786_a23b_470f_bf3d_2a9cd51ff2ee.slice/crio-ea9fdae01c1fca26263d97a02efc95ef1e6c4a9b4066abd95051e6543c84f4e9 WatchSource:0}: Error finding container ea9fdae01c1fca26263d97a02efc95ef1e6c4a9b4066abd95051e6543c84f4e9: Status 404 returned error can't find the container with id ea9fdae01c1fca26263d97a02efc95ef1e6c4a9b4066abd95051e6543c84f4e9
Jul 08 16:37:08.682259 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:08.682222    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt" event={"ID":"a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee","Type":"ContainerStarted","Data":"ea9fdae01c1fca26263d97a02efc95ef1e6c4a9b4066abd95051e6543c84f4e9"}
Jul 08 16:37:11.696057 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:11.696028    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt" event={"ID":"a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee","Type":"ContainerStarted","Data":"58cade6db02b2ffc534854a8c2d1edf0840fd6ff693a0b793fb7e253d2abe82d"}
Jul 08 16:37:11.696370 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:11.696093    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt" event={"ID":"a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee","Type":"ContainerStarted","Data":"278530f39019835fe406d80de7d0bddbe221918b1ce2d77af5f013e46e8d0d10"}
Jul 08 16:37:11.696370 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:11.696116    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt" event={"ID":"a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee","Type":"ContainerStarted","Data":"fa766168b425bb248929bc977745a57883e7cf39ca800f7ba6e1bd2509ad9762"}
Jul 08 16:37:11.721539 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:11.721495    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-646c8d68d4-kd7vt" podStartSLOduration=1.627147464 podStartE2EDuration="4.72147967s" totalImagesPullingTime="3.094332206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:37:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:37:11.71957372 +0000 UTC m=+168.432935171" watchObservedRunningTime="2026-07-08 16:37:11.72147967 +0000 UTC m=+168.434841099"
Jul 08 16:37:12.619474 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:12.619433    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:17.191943 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:17.191855    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:17.301328 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:17.301295    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:18.074873 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:18.074839    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:18.149795 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:18.149767    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:22.473266 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.473227    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cd65d986f-5vl7z"]
Jul 08 16:37:22.477659 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.477639    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd65d986f-5vl7z"
Jul 08 16:37:22.480098 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.480070    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 08 16:37:22.480185 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.480141    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 08 16:37:22.480226 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.480203    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 08 16:37:22.481060 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.481032    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 08 16:37:22.481060 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.481052    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 08 16:37:22.481216 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.481157    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 08 16:37:22.481216 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.481176    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 08 16:37:22.481412 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.481393    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kfjdt\""
Jul 08 16:37:22.485893 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.485872    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 08 16:37:22.491317 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.491298    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cd65d986f-5vl7z"]
Jul 08 16:37:22.597200 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.597162    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe1b9009-6574-409f-8e18-7a7eb8146dc5-trusted-ca-bundle\") pod \"console-6cd65d986f-5vl7z\" (UID: \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\") " pod="openshift-console/console-6cd65d986f-5vl7z"
Jul 08 16:37:22.597200 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.597195    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe1b9009-6574-409f-8e18-7a7eb8146dc5-oauth-serving-cert\") pod \"console-6cd65d986f-5vl7z\" (UID: \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\") " pod="openshift-console/console-6cd65d986f-5vl7z"
Jul 08 16:37:22.597400 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.597213    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe1b9009-6574-409f-8e18-7a7eb8146dc5-console-config\") pod \"console-6cd65d986f-5vl7z\" (UID: \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\") " pod="openshift-console/console-6cd65d986f-5vl7z"
Jul 08 16:37:22.597400 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.597305    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe1b9009-6574-409f-8e18-7a7eb8146dc5-service-ca\") pod \"console-6cd65d986f-5vl7z\" (UID: \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\") " pod="openshift-console/console-6cd65d986f-5vl7z"
Jul 08 16:37:22.597400 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.597365    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j86b4\" (UniqueName: \"kubernetes.io/projected/fe1b9009-6574-409f-8e18-7a7eb8146dc5-kube-api-access-j86b4\") pod \"console-6cd65d986f-5vl7z\" (UID: \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\") " pod="openshift-console/console-6cd65d986f-5vl7z"
Jul 08 16:37:22.597400 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.597395    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe1b9009-6574-409f-8e18-7a7eb8146dc5-console-serving-cert\") pod \"console-6cd65d986f-5vl7z\" (UID: \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\") " pod="openshift-console/console-6cd65d986f-5vl7z"
Jul 08 16:37:22.597524 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.597422    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe1b9009-6574-409f-8e18-7a7eb8146dc5-console-oauth-config\") pod \"console-6cd65d986f-5vl7z\" (UID: \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\") " pod="openshift-console/console-6cd65d986f-5vl7z"
Jul 08 16:37:22.698713 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.698648    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe1b9009-6574-409f-8e18-7a7eb8146dc5-trusted-ca-bundle\") pod \"console-6cd65d986f-5vl7z\" (UID: \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\") " pod="openshift-console/console-6cd65d986f-5vl7z"
Jul 08 16:37:22.698713 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.698718    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe1b9009-6574-409f-8e18-7a7eb8146dc5-oauth-serving-cert\") pod \"console-6cd65d986f-5vl7z\" (UID: \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\") " pod="openshift-console/console-6cd65d986f-5vl7z"
Jul 08 16:37:22.698861 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.698735    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe1b9009-6574-409f-8e18-7a7eb8146dc5-console-config\") pod \"console-6cd65d986f-5vl7z\" (UID: \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\") " pod="openshift-console/console-6cd65d986f-5vl7z"
Jul 08 16:37:22.698861 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.698772    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe1b9009-6574-409f-8e18-7a7eb8146dc5-service-ca\") pod \"console-6cd65d986f-5vl7z\" (UID: \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\") " pod="openshift-console/console-6cd65d986f-5vl7z"
Jul 08 16:37:22.698861 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.698797    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j86b4\" (UniqueName: \"kubernetes.io/projected/fe1b9009-6574-409f-8e18-7a7eb8146dc5-kube-api-access-j86b4\") pod \"console-6cd65d986f-5vl7z\" (UID: \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\") " pod="openshift-console/console-6cd65d986f-5vl7z"
Jul 08 16:37:22.698861 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.698817    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe1b9009-6574-409f-8e18-7a7eb8146dc5-console-serving-cert\") pod \"console-6cd65d986f-5vl7z\" (UID: \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\") " pod="openshift-console/console-6cd65d986f-5vl7z"
Jul 08 16:37:22.698861 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.698838    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe1b9009-6574-409f-8e18-7a7eb8146dc5-console-oauth-config\") pod \"console-6cd65d986f-5vl7z\" (UID: \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\") " pod="openshift-console/console-6cd65d986f-5vl7z"
Jul 08 16:37:22.699470 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.699439    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe1b9009-6574-409f-8e18-7a7eb8146dc5-oauth-serving-cert\") pod \"console-6cd65d986f-5vl7z\" (UID: \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\") " pod="openshift-console/console-6cd65d986f-5vl7z"
Jul 08 16:37:22.699587 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.699519    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe1b9009-6574-409f-8e18-7a7eb8146dc5-console-config\") pod \"console-6cd65d986f-5vl7z\" (UID: \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\") " pod="openshift-console/console-6cd65d986f-5vl7z"
Jul 08 16:37:22.699648 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.699585    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe1b9009-6574-409f-8e18-7a7eb8146dc5-service-ca\") pod \"console-6cd65d986f-5vl7z\" (UID: \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\") " pod="openshift-console/console-6cd65d986f-5vl7z"
Jul 08 16:37:22.699741 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.699704    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe1b9009-6574-409f-8e18-7a7eb8146dc5-trusted-ca-bundle\") pod \"console-6cd65d986f-5vl7z\" (UID: \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\") " pod="openshift-console/console-6cd65d986f-5vl7z"
Jul 08 16:37:22.701187 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.701164    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe1b9009-6574-409f-8e18-7a7eb8146dc5-console-oauth-config\") pod \"console-6cd65d986f-5vl7z\" (UID: \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\") " pod="openshift-console/console-6cd65d986f-5vl7z"
Jul 08 16:37:22.701409 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.701389    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe1b9009-6574-409f-8e18-7a7eb8146dc5-console-serving-cert\") pod \"console-6cd65d986f-5vl7z\" (UID: \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\") " pod="openshift-console/console-6cd65d986f-5vl7z"
Jul 08 16:37:22.708162 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.708142    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j86b4\" (UniqueName: \"kubernetes.io/projected/fe1b9009-6574-409f-8e18-7a7eb8146dc5-kube-api-access-j86b4\") pod \"console-6cd65d986f-5vl7z\" (UID: \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\") " pod="openshift-console/console-6cd65d986f-5vl7z"
Jul 08 16:37:22.787443 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.787389    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd65d986f-5vl7z"
Jul 08 16:37:22.918875 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:22.918850    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cd65d986f-5vl7z"]
Jul 08 16:37:22.921060 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:37:22.921034    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe1b9009_6574_409f_8e18_7a7eb8146dc5.slice/crio-ffcde9be8254bf69cc218b4cd39ce5fcabe5f3453bc96b4ccb50d4963e7be6e7 WatchSource:0}: Error finding container ffcde9be8254bf69cc218b4cd39ce5fcabe5f3453bc96b4ccb50d4963e7be6e7: Status 404 returned error can't find the container with id ffcde9be8254bf69cc218b4cd39ce5fcabe5f3453bc96b4ccb50d4963e7be6e7
Jul 08 16:37:23.738338 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:23.738300    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd65d986f-5vl7z" event={"ID":"fe1b9009-6574-409f-8e18-7a7eb8146dc5","Type":"ContainerStarted","Data":"18e0d13489b65332ae264bd50af684faee2ebddbcb7d1c7b5b5243b5bb4494a5"}
Jul 08 16:37:23.738774 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:23.738391    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd65d986f-5vl7z" event={"ID":"fe1b9009-6574-409f-8e18-7a7eb8146dc5","Type":"ContainerStarted","Data":"ffcde9be8254bf69cc218b4cd39ce5fcabe5f3453bc96b4ccb50d4963e7be6e7"}
Jul 08 16:37:23.759129 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:23.759088    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6cd65d986f-5vl7z" podStartSLOduration=1.75907309 podStartE2EDuration="1.75907309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:37:23.757468508 +0000 UTC m=+180.470829937" watchObservedRunningTime="2026-07-08 16:37:23.75907309 +0000 UTC m=+180.472434517"
Jul 08 16:37:30.114320 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:30.114284    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:30.302801 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:30.302769    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:32.787750 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:32.787720    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6cd65d986f-5vl7z"
Jul 08 16:37:32.788146 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:32.787763    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cd65d986f-5vl7z"
Jul 08 16:37:32.792442 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:32.792418    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6cd65d986f-5vl7z"
Jul 08 16:37:33.779115 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:33.779088    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6cd65d986f-5vl7z"
Jul 08 16:37:33.973425 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:33.973386    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:34.080420 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:34.080342    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:34.150423 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:34.150396    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:37.470321 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:37.470287    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:37.533627 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:37.533596    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:37:43.456006 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:37:43.455974    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:38:08.804227 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:08.804185    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:38:08.866804 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:08.866771    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:38:22.755861 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:22.755822    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:38:32.278049 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:32.278005    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:38:32.345559 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:32.345526    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:38:32.616123 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:32.616034    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:38:32.717596 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:32.717561    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:38:32.816711 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:32.816660    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:38:32.991269 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:32.991233    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:38:33.656579 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:33.656543    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx"]
Jul 08 16:38:33.659909 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:33.659893    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx"
Jul 08 16:38:33.662393 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:33.662373    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 08 16:38:33.663256 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:33.663240    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mlhdn\""
Jul 08 16:38:33.663256 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:33.663249    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 08 16:38:33.669635 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:33.669541    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx"]
Jul 08 16:38:33.750440 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:33.750412    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/086306f1-bd8c-4a98-960d-fdba72759b62-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx\" (UID: \"086306f1-bd8c-4a98-960d-fdba72759b62\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx"
Jul 08 16:38:33.750617 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:33.750488    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/086306f1-bd8c-4a98-960d-fdba72759b62-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx\" (UID: \"086306f1-bd8c-4a98-960d-fdba72759b62\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx"
Jul 08 16:38:33.750617 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:33.750521    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98phn\" (UniqueName: \"kubernetes.io/projected/086306f1-bd8c-4a98-960d-fdba72759b62-kube-api-access-98phn\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx\" (UID: \"086306f1-bd8c-4a98-960d-fdba72759b62\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx"
Jul 08 16:38:33.851242 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:33.851207    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/086306f1-bd8c-4a98-960d-fdba72759b62-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx\" (UID: \"086306f1-bd8c-4a98-960d-fdba72759b62\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx"
Jul 08 16:38:33.851242 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:33.851251    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98phn\" (UniqueName: \"kubernetes.io/projected/086306f1-bd8c-4a98-960d-fdba72759b62-kube-api-access-98phn\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx\" (UID: \"086306f1-bd8c-4a98-960d-fdba72759b62\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx"
Jul 08 16:38:33.851519 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:33.851284    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/086306f1-bd8c-4a98-960d-fdba72759b62-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx\" (UID: \"086306f1-bd8c-4a98-960d-fdba72759b62\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx"
Jul 08 16:38:33.851706 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:33.851654    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/086306f1-bd8c-4a98-960d-fdba72759b62-util\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx\" (UID: \"086306f1-bd8c-4a98-960d-fdba72759b62\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx"
Jul 08 16:38:33.851706 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:33.851669    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/086306f1-bd8c-4a98-960d-fdba72759b62-bundle\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx\" (UID: \"086306f1-bd8c-4a98-960d-fdba72759b62\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx"
Jul 08 16:38:33.861459 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:33.861419    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98phn\" (UniqueName: \"kubernetes.io/projected/086306f1-bd8c-4a98-960d-fdba72759b62-kube-api-access-98phn\") pod \"74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx\" (UID: \"086306f1-bd8c-4a98-960d-fdba72759b62\") " pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx"
Jul 08 16:38:33.968653 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:33.968614    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx"
Jul 08 16:38:34.113569 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:34.113527    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx"]
Jul 08 16:38:34.118394 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:38:34.118335    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod086306f1_bd8c_4a98_960d_fdba72759b62.slice/crio-ae552d6864256ae0ddbd01780cdbf59d4a463416dd45a2e48cb967612387638e WatchSource:0}: Error finding container ae552d6864256ae0ddbd01780cdbf59d4a463416dd45a2e48cb967612387638e: Status 404 returned error can't find the container with id ae552d6864256ae0ddbd01780cdbf59d4a463416dd45a2e48cb967612387638e
Jul 08 16:38:34.991670 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:34.991635    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx" event={"ID":"086306f1-bd8c-4a98-960d-fdba72759b62","Type":"ContainerStarted","Data":"ae552d6864256ae0ddbd01780cdbf59d4a463416dd45a2e48cb967612387638e"}
Jul 08 16:38:42.023910 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:42.023874    2565 generic.go:356] "Generic (PLEG): container finished" podID="086306f1-bd8c-4a98-960d-fdba72759b62" containerID="83331a5dc3ad6b523da11af3cbc3bcde456b8dbf5c33aebd91df63f7f974fba5" exitCode=0
Jul 08 16:38:42.024283 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:42.023944    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx" event={"ID":"086306f1-bd8c-4a98-960d-fdba72759b62","Type":"ContainerDied","Data":"83331a5dc3ad6b523da11af3cbc3bcde456b8dbf5c33aebd91df63f7f974fba5"}
Jul 08 16:38:44.034497 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:44.034459    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx" event={"ID":"086306f1-bd8c-4a98-960d-fdba72759b62","Type":"ContainerStarted","Data":"18ae3663c00c33efe46e09005acc7557f8365223e3d85c2f16ce2f5f675246ea"}
Jul 08 16:38:45.040344 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:45.040259    2565 generic.go:356] "Generic (PLEG): container finished" podID="086306f1-bd8c-4a98-960d-fdba72759b62" containerID="18ae3663c00c33efe46e09005acc7557f8365223e3d85c2f16ce2f5f675246ea" exitCode=0
Jul 08 16:38:45.040344 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:45.040314    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx" event={"ID":"086306f1-bd8c-4a98-960d-fdba72759b62","Type":"ContainerDied","Data":"18ae3663c00c33efe46e09005acc7557f8365223e3d85c2f16ce2f5f675246ea"}
Jul 08 16:38:53.075723 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:53.075618    2565 generic.go:356] "Generic (PLEG): container finished" podID="086306f1-bd8c-4a98-960d-fdba72759b62" containerID="2e0f583142d8e490e1c55ccfcad4a86b0f1f418e6e0318bab4d6a1b901aa776a" exitCode=0
Jul 08 16:38:53.075723 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:53.075709    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx" event={"ID":"086306f1-bd8c-4a98-960d-fdba72759b62","Type":"ContainerDied","Data":"2e0f583142d8e490e1c55ccfcad4a86b0f1f418e6e0318bab4d6a1b901aa776a"}
Jul 08 16:38:54.208508 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:54.208483    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx"
Jul 08 16:38:54.337223 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:54.337095    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98phn\" (UniqueName: \"kubernetes.io/projected/086306f1-bd8c-4a98-960d-fdba72759b62-kube-api-access-98phn\") pod \"086306f1-bd8c-4a98-960d-fdba72759b62\" (UID: \"086306f1-bd8c-4a98-960d-fdba72759b62\") "
Jul 08 16:38:54.337223 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:54.337197    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/086306f1-bd8c-4a98-960d-fdba72759b62-util\") pod \"086306f1-bd8c-4a98-960d-fdba72759b62\" (UID: \"086306f1-bd8c-4a98-960d-fdba72759b62\") "
Jul 08 16:38:54.337223 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:54.337228    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/086306f1-bd8c-4a98-960d-fdba72759b62-bundle\") pod \"086306f1-bd8c-4a98-960d-fdba72759b62\" (UID: \"086306f1-bd8c-4a98-960d-fdba72759b62\") "
Jul 08 16:38:54.338010 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:54.337986    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/086306f1-bd8c-4a98-960d-fdba72759b62-bundle" (OuterVolumeSpecName: "bundle") pod "086306f1-bd8c-4a98-960d-fdba72759b62" (UID: "086306f1-bd8c-4a98-960d-fdba72759b62"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:38:54.339555 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:54.339529    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/086306f1-bd8c-4a98-960d-fdba72759b62-kube-api-access-98phn" (OuterVolumeSpecName: "kube-api-access-98phn") pod "086306f1-bd8c-4a98-960d-fdba72759b62" (UID: "086306f1-bd8c-4a98-960d-fdba72759b62"). InnerVolumeSpecName "kube-api-access-98phn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:38:54.341859 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:54.341833    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/086306f1-bd8c-4a98-960d-fdba72759b62-util" (OuterVolumeSpecName: "util") pod "086306f1-bd8c-4a98-960d-fdba72759b62" (UID: "086306f1-bd8c-4a98-960d-fdba72759b62"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:38:54.438503 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:54.438460    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/086306f1-bd8c-4a98-960d-fdba72759b62-bundle\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:38:54.438503 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:54.438492    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98phn\" (UniqueName: \"kubernetes.io/projected/086306f1-bd8c-4a98-960d-fdba72759b62-kube-api-access-98phn\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:38:54.438503 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:54.438501    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/086306f1-bd8c-4a98-960d-fdba72759b62-util\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:38:55.084652 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:55.084628    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx"
Jul 08 16:38:55.084652 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:55.084642    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/74e94af6ec69867d65f1d90648440bdc4f0845d5e32c4818a460ebb589lbwgx" event={"ID":"086306f1-bd8c-4a98-960d-fdba72759b62","Type":"ContainerDied","Data":"ae552d6864256ae0ddbd01780cdbf59d4a463416dd45a2e48cb967612387638e"}
Jul 08 16:38:55.084870 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:55.084696    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae552d6864256ae0ddbd01780cdbf59d4a463416dd45a2e48cb967612387638e"
Jul 08 16:38:59.970807 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:38:59.970765    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:00.070831 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:00.070790    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:01.602541 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:01.602507    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-9xllg"]
Jul 08 16:39:01.602958 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:01.602817    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="086306f1-bd8c-4a98-960d-fdba72759b62" containerName="util"
Jul 08 16:39:01.602958 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:01.602829    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="086306f1-bd8c-4a98-960d-fdba72759b62" containerName="util"
Jul 08 16:39:01.602958 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:01.602843    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="086306f1-bd8c-4a98-960d-fdba72759b62" containerName="extract"
Jul 08 16:39:01.602958 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:01.602848    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="086306f1-bd8c-4a98-960d-fdba72759b62" containerName="extract"
Jul 08 16:39:01.602958 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:01.602859    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="086306f1-bd8c-4a98-960d-fdba72759b62" containerName="pull"
Jul 08 16:39:01.602958 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:01.602864    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="086306f1-bd8c-4a98-960d-fdba72759b62" containerName="pull"
Jul 08 16:39:01.602958 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:01.602919    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="086306f1-bd8c-4a98-960d-fdba72759b62" containerName="extract"
Jul 08 16:39:01.607116 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:01.607095    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-9xllg"
Jul 08 16:39:01.609823 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:01.609801    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-gzqg6\""
Jul 08 16:39:01.609823 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:01.609816    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 08 16:39:01.609997 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:01.609805    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 08 16:39:01.610184 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:01.610172    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 08 16:39:01.619466 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:01.619441    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-9xllg"]
Jul 08 16:39:01.701742 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:01.701706    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46bmn\" (UniqueName: \"kubernetes.io/projected/8ea777bb-9b89-4577-a8a8-f7ca84e72c07-kube-api-access-46bmn\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-9xllg\" (UID: \"8ea777bb-9b89-4577-a8a8-f7ca84e72c07\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-9xllg"
Jul 08 16:39:01.701917 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:01.701777    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/8ea777bb-9b89-4577-a8a8-f7ca84e72c07-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-9xllg\" (UID: \"8ea777bb-9b89-4577-a8a8-f7ca84e72c07\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-9xllg"
Jul 08 16:39:01.802724 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:01.802653    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/8ea777bb-9b89-4577-a8a8-f7ca84e72c07-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-9xllg\" (UID: \"8ea777bb-9b89-4577-a8a8-f7ca84e72c07\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-9xllg"
Jul 08 16:39:01.802912 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:01.802769    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46bmn\" (UniqueName: \"kubernetes.io/projected/8ea777bb-9b89-4577-a8a8-f7ca84e72c07-kube-api-access-46bmn\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-9xllg\" (UID: \"8ea777bb-9b89-4577-a8a8-f7ca84e72c07\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-9xllg"
Jul 08 16:39:01.805050 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:01.805030    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/8ea777bb-9b89-4577-a8a8-f7ca84e72c07-certificates\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-9xllg\" (UID: \"8ea777bb-9b89-4577-a8a8-f7ca84e72c07\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-9xllg"
Jul 08 16:39:01.813372 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:01.813345    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46bmn\" (UniqueName: \"kubernetes.io/projected/8ea777bb-9b89-4577-a8a8-f7ca84e72c07-kube-api-access-46bmn\") pod \"custom-metrics-autoscaler-operator-75f7c5f8f-9xllg\" (UID: \"8ea777bb-9b89-4577-a8a8-f7ca84e72c07\") " pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-9xllg"
Jul 08 16:39:01.917488 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:01.917387    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-9xllg"
Jul 08 16:39:02.060052 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:02.060021    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-9xllg"]
Jul 08 16:39:02.063610 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:39:02.063578    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea777bb_9b89_4577_a8a8_f7ca84e72c07.slice/crio-e929e69f14a50d17326099fb60705b5e3cb04486001c8537f32040ed43ca14eb WatchSource:0}: Error finding container e929e69f14a50d17326099fb60705b5e3cb04486001c8537f32040ed43ca14eb: Status 404 returned error can't find the container with id e929e69f14a50d17326099fb60705b5e3cb04486001c8537f32040ed43ca14eb
Jul 08 16:39:02.113108 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:02.113067    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-9xllg" event={"ID":"8ea777bb-9b89-4577-a8a8-f7ca84e72c07","Type":"ContainerStarted","Data":"e929e69f14a50d17326099fb60705b5e3cb04486001c8537f32040ed43ca14eb"}
Jul 08 16:39:08.845928 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:08.845893    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:09.126178 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:09.126074    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:09.149214 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:09.149172    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-68bff59c-tm5sf"]
Jul 08 16:39:09.151514 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:09.151491    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-9xllg"
Jul 08 16:39:09.151654 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:09.151524    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-9xllg" event={"ID":"8ea777bb-9b89-4577-a8a8-f7ca84e72c07","Type":"ContainerStarted","Data":"9fb3af2b4fc290f7013546689d8b43fdda16421f5582334c0546536d7129b316"}
Jul 08 16:39:09.151654 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:09.151608    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-tm5sf"
Jul 08 16:39:09.154054 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:09.154012    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 08 16:39:09.154054 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:09.154023    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-6nfsb\""
Jul 08 16:39:09.154206 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:09.154093    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 08 16:39:09.165750 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:09.165718    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-tm5sf"]
Jul 08 16:39:09.190465 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:09.190409    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-9xllg" podStartSLOduration=1.8747009239999999 podStartE2EDuration="8.190392529s" totalImagesPullingTime="6.315691605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:39:09.188360182 +0000 UTC m=+285.901721630" watchObservedRunningTime="2026-07-08 16:39:09.190392529 +0000 UTC m=+285.903753955"
Jul 08 16:39:09.266326 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:09.266276    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4v2z\" (UniqueName: \"kubernetes.io/projected/834a6c05-f2ae-4eed-ab18-c6b47eecbdb6-kube-api-access-v4v2z\") pod \"keda-operator-68bff59c-tm5sf\" (UID: \"834a6c05-f2ae-4eed-ab18-c6b47eecbdb6\") " pod="openshift-keda/keda-operator-68bff59c-tm5sf"
Jul 08 16:39:09.266525 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:09.266335    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/834a6c05-f2ae-4eed-ab18-c6b47eecbdb6-cabundle0\") pod \"keda-operator-68bff59c-tm5sf\" (UID: \"834a6c05-f2ae-4eed-ab18-c6b47eecbdb6\") " pod="openshift-keda/keda-operator-68bff59c-tm5sf"
Jul 08 16:39:09.266525 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:09.266374    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/834a6c05-f2ae-4eed-ab18-c6b47eecbdb6-certificates\") pod \"keda-operator-68bff59c-tm5sf\" (UID: \"834a6c05-f2ae-4eed-ab18-c6b47eecbdb6\") " pod="openshift-keda/keda-operator-68bff59c-tm5sf"
Jul 08 16:39:09.367549 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:09.367508    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4v2z\" (UniqueName: \"kubernetes.io/projected/834a6c05-f2ae-4eed-ab18-c6b47eecbdb6-kube-api-access-v4v2z\") pod \"keda-operator-68bff59c-tm5sf\" (UID: \"834a6c05-f2ae-4eed-ab18-c6b47eecbdb6\") " pod="openshift-keda/keda-operator-68bff59c-tm5sf"
Jul 08 16:39:09.367549 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:09.367547    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/834a6c05-f2ae-4eed-ab18-c6b47eecbdb6-cabundle0\") pod \"keda-operator-68bff59c-tm5sf\" (UID: \"834a6c05-f2ae-4eed-ab18-c6b47eecbdb6\") " pod="openshift-keda/keda-operator-68bff59c-tm5sf"
Jul 08 16:39:09.367813 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:09.367568    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/834a6c05-f2ae-4eed-ab18-c6b47eecbdb6-certificates\") pod \"keda-operator-68bff59c-tm5sf\" (UID: \"834a6c05-f2ae-4eed-ab18-c6b47eecbdb6\") " pod="openshift-keda/keda-operator-68bff59c-tm5sf"
Jul 08 16:39:09.367813 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:39:09.367660    2565 secret.go:281] references non-existent secret key: ca.crt
Jul 08 16:39:09.367813 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:39:09.367695    2565 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 08 16:39:09.367813 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:39:09.367706    2565 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-68bff59c-tm5sf: references non-existent secret key: ca.crt
Jul 08 16:39:09.367813 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:39:09.367768    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/834a6c05-f2ae-4eed-ab18-c6b47eecbdb6-certificates podName:834a6c05-f2ae-4eed-ab18-c6b47eecbdb6 nodeName:}" failed. No retries permitted until 2026-07-08 16:39:09.867751866 +0000 UTC m=+286.581113275 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/834a6c05-f2ae-4eed-ab18-c6b47eecbdb6-certificates") pod "keda-operator-68bff59c-tm5sf" (UID: "834a6c05-f2ae-4eed-ab18-c6b47eecbdb6") : references non-existent secret key: ca.crt
Jul 08 16:39:09.368189 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:09.368168    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/834a6c05-f2ae-4eed-ab18-c6b47eecbdb6-cabundle0\") pod \"keda-operator-68bff59c-tm5sf\" (UID: \"834a6c05-f2ae-4eed-ab18-c6b47eecbdb6\") " pod="openshift-keda/keda-operator-68bff59c-tm5sf"
Jul 08 16:39:09.377545 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:09.377476    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4v2z\" (UniqueName: \"kubernetes.io/projected/834a6c05-f2ae-4eed-ab18-c6b47eecbdb6-kube-api-access-v4v2z\") pod \"keda-operator-68bff59c-tm5sf\" (UID: \"834a6c05-f2ae-4eed-ab18-c6b47eecbdb6\") " pod="openshift-keda/keda-operator-68bff59c-tm5sf"
Jul 08 16:39:09.426886 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:09.426851    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:09.587720 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:09.587690    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:09.784909 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:09.784862    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:09.870758 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:09.870727    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/834a6c05-f2ae-4eed-ab18-c6b47eecbdb6-certificates\") pod \"keda-operator-68bff59c-tm5sf\" (UID: \"834a6c05-f2ae-4eed-ab18-c6b47eecbdb6\") " pod="openshift-keda/keda-operator-68bff59c-tm5sf"
Jul 08 16:39:09.873595 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:09.873566    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/834a6c05-f2ae-4eed-ab18-c6b47eecbdb6-certificates\") pod \"keda-operator-68bff59c-tm5sf\" (UID: \"834a6c05-f2ae-4eed-ab18-c6b47eecbdb6\") " pod="openshift-keda/keda-operator-68bff59c-tm5sf"
Jul 08 16:39:09.913300 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:09.913268    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:09.987329 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:09.987291    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-vxzfp"]
Jul 08 16:39:09.989753 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:09.989731    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-vxzfp"
Jul 08 16:39:09.992033 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:09.992013    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 08 16:39:10.001318 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:10.001294    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-vxzfp"]
Jul 08 16:39:10.021427 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:10.021394    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:10.069451 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:10.069375    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-68bff59c-tm5sf"
Jul 08 16:39:10.072439 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:10.072418    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7f55d1d2-4a39-4924-bfb1-478068d545e7-certificates\") pod \"keda-admission-5fbd5c4644-vxzfp\" (UID: \"7f55d1d2-4a39-4924-bfb1-478068d545e7\") " pod="openshift-keda/keda-admission-5fbd5c4644-vxzfp"
Jul 08 16:39:10.072557 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:10.072466    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzgkd\" (UniqueName: \"kubernetes.io/projected/7f55d1d2-4a39-4924-bfb1-478068d545e7-kube-api-access-mzgkd\") pod \"keda-admission-5fbd5c4644-vxzfp\" (UID: \"7f55d1d2-4a39-4924-bfb1-478068d545e7\") " pod="openshift-keda/keda-admission-5fbd5c4644-vxzfp"
Jul 08 16:39:10.118255 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:10.118218    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:10.173185 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:10.173151    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7f55d1d2-4a39-4924-bfb1-478068d545e7-certificates\") pod \"keda-admission-5fbd5c4644-vxzfp\" (UID: \"7f55d1d2-4a39-4924-bfb1-478068d545e7\") " pod="openshift-keda/keda-admission-5fbd5c4644-vxzfp"
Jul 08 16:39:10.173344 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:10.173207    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzgkd\" (UniqueName: \"kubernetes.io/projected/7f55d1d2-4a39-4924-bfb1-478068d545e7-kube-api-access-mzgkd\") pod \"keda-admission-5fbd5c4644-vxzfp\" (UID: \"7f55d1d2-4a39-4924-bfb1-478068d545e7\") " pod="openshift-keda/keda-admission-5fbd5c4644-vxzfp"
Jul 08 16:39:10.173387 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:39:10.173359    2565 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 08 16:39:10.173432 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:39:10.173388    2565 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-5fbd5c4644-vxzfp: secret "keda-admission-webhooks-certs" not found
Jul 08 16:39:10.173485 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:39:10.173457    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7f55d1d2-4a39-4924-bfb1-478068d545e7-certificates podName:7f55d1d2-4a39-4924-bfb1-478068d545e7 nodeName:}" failed. No retries permitted until 2026-07-08 16:39:10.673435961 +0000 UTC m=+287.386797382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/7f55d1d2-4a39-4924-bfb1-478068d545e7-certificates") pod "keda-admission-5fbd5c4644-vxzfp" (UID: "7f55d1d2-4a39-4924-bfb1-478068d545e7") : secret "keda-admission-webhooks-certs" not found
Jul 08 16:39:10.184201 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:10.184167    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzgkd\" (UniqueName: \"kubernetes.io/projected/7f55d1d2-4a39-4924-bfb1-478068d545e7-kube-api-access-mzgkd\") pod \"keda-admission-5fbd5c4644-vxzfp\" (UID: \"7f55d1d2-4a39-4924-bfb1-478068d545e7\") " pod="openshift-keda/keda-admission-5fbd5c4644-vxzfp"
Jul 08 16:39:10.212742 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:10.212707    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-68bff59c-tm5sf"]
Jul 08 16:39:10.217519 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:39:10.217492    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod834a6c05_f2ae_4eed_ab18_c6b47eecbdb6.slice/crio-de8e1b03df2387cabfd7e6b347480e7b81b0f750c5ac9f0a19e16f576caf5468 WatchSource:0}: Error finding container de8e1b03df2387cabfd7e6b347480e7b81b0f750c5ac9f0a19e16f576caf5468: Status 404 returned error can't find the container with id de8e1b03df2387cabfd7e6b347480e7b81b0f750c5ac9f0a19e16f576caf5468
Jul 08 16:39:10.253601 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:10.253571    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:10.502811 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:10.502775    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:10.568082 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:10.568041    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:10.677520 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:10.677481    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7f55d1d2-4a39-4924-bfb1-478068d545e7-certificates\") pod \"keda-admission-5fbd5c4644-vxzfp\" (UID: \"7f55d1d2-4a39-4924-bfb1-478068d545e7\") " pod="openshift-keda/keda-admission-5fbd5c4644-vxzfp"
Jul 08 16:39:10.679954 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:10.679932    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/7f55d1d2-4a39-4924-bfb1-478068d545e7-certificates\") pod \"keda-admission-5fbd5c4644-vxzfp\" (UID: \"7f55d1d2-4a39-4924-bfb1-478068d545e7\") " pod="openshift-keda/keda-admission-5fbd5c4644-vxzfp"
Jul 08 16:39:10.702761 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:10.702724    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:10.802985 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:10.802896    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:10.900521 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:10.900487    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-5fbd5c4644-vxzfp"
Jul 08 16:39:11.028613 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:11.028589    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-5fbd5c4644-vxzfp"]
Jul 08 16:39:11.031191 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:39:11.031163    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f55d1d2_4a39_4924_bfb1_478068d545e7.slice/crio-5dcc91456ebd29377b2b294e1ec83ff889dcf76f66702b306994eaf08672944c WatchSource:0}: Error finding container 5dcc91456ebd29377b2b294e1ec83ff889dcf76f66702b306994eaf08672944c: Status 404 returned error can't find the container with id 5dcc91456ebd29377b2b294e1ec83ff889dcf76f66702b306994eaf08672944c
Jul 08 16:39:11.161402 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:11.161323    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-tm5sf" event={"ID":"834a6c05-f2ae-4eed-ab18-c6b47eecbdb6","Type":"ContainerStarted","Data":"de8e1b03df2387cabfd7e6b347480e7b81b0f750c5ac9f0a19e16f576caf5468"}
Jul 08 16:39:11.162553 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:11.162521    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-vxzfp" event={"ID":"7f55d1d2-4a39-4924-bfb1-478068d545e7","Type":"ContainerStarted","Data":"5dcc91456ebd29377b2b294e1ec83ff889dcf76f66702b306994eaf08672944c"}
Jul 08 16:39:14.180149 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:14.180113    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-68bff59c-tm5sf" event={"ID":"834a6c05-f2ae-4eed-ab18-c6b47eecbdb6","Type":"ContainerStarted","Data":"6d1d523693c01e56a65247f4b6b2ccc9a4e851f654dbbf12e67114e3505f4ddd"}
Jul 08 16:39:14.180633 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:14.180212    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-68bff59c-tm5sf"
Jul 08 16:39:14.181713 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:14.181687    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-5fbd5c4644-vxzfp" event={"ID":"7f55d1d2-4a39-4924-bfb1-478068d545e7","Type":"ContainerStarted","Data":"95bd23d4356ae55a8c8d326938d0c15a4324cbad2a2c2515996cd2877f300ebe"}
Jul 08 16:39:14.181814 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:14.181798    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-5fbd5c4644-vxzfp"
Jul 08 16:39:14.200648 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:14.200595    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-68bff59c-tm5sf" podStartSLOduration=1.591646269 podStartE2EDuration="5.200576218s" totalImagesPullingTime="3.608929949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:39:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:39:14.19974071 +0000 UTC m=+290.913102139" watchObservedRunningTime="2026-07-08 16:39:14.200576218 +0000 UTC m=+290.913937645"
Jul 08 16:39:14.221536 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:14.221481    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-5fbd5c4644-vxzfp" podStartSLOduration=2.429798665 podStartE2EDuration="5.221462795s" totalImagesPullingTime="2.79166413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:39:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:39:14.220764118 +0000 UTC m=+290.934125546" watchObservedRunningTime="2026-07-08 16:39:14.221462795 +0000 UTC m=+290.934824222"
Jul 08 16:39:16.154616 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:16.154574    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:23.662116 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:23.662065    2565 kubelet.go:1648] "Image garbage collection succeeded"
Jul 08 16:39:23.755546 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:23.755507    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:23.827252 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:23.827087    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:29.649534 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:29.649495    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:39:30.159382 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:30.159345    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-75f7c5f8f-9xllg"
Jul 08 16:39:35.188222 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:35.188192    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-68bff59c-tm5sf"
Jul 08 16:39:35.188701 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:39:35.188346    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-5fbd5c4644-vxzfp"
Jul 08 16:40:19.605205 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:19.605122    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:40:19.674577 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:19.674540    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-687b7449bc-5hzqb"]
Jul 08 16:40:19.676959 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:19.676943    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-687b7449bc-5hzqb"
Jul 08 16:40:19.679815 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:19.679794    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-cfz9g\""
Jul 08 16:40:19.679944 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:19.679799    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 08 16:40:19.680778 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:19.680752    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 08 16:40:19.680891 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:19.680831    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 08 16:40:19.691016 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:19.690994    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-687b7449bc-5hzqb"]
Jul 08 16:40:19.777395 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:19.777350    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4e253de-313d-4de6-8e27-cc24d13b355c-cert\") pod \"kserve-controller-manager-687b7449bc-5hzqb\" (UID: \"b4e253de-313d-4de6-8e27-cc24d13b355c\") " pod="kserve/kserve-controller-manager-687b7449bc-5hzqb"
Jul 08 16:40:19.777577 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:19.777408    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9t58\" (UniqueName: \"kubernetes.io/projected/b4e253de-313d-4de6-8e27-cc24d13b355c-kube-api-access-j9t58\") pod \"kserve-controller-manager-687b7449bc-5hzqb\" (UID: \"b4e253de-313d-4de6-8e27-cc24d13b355c\") " pod="kserve/kserve-controller-manager-687b7449bc-5hzqb"
Jul 08 16:40:19.878559 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:19.878440    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4e253de-313d-4de6-8e27-cc24d13b355c-cert\") pod \"kserve-controller-manager-687b7449bc-5hzqb\" (UID: \"b4e253de-313d-4de6-8e27-cc24d13b355c\") " pod="kserve/kserve-controller-manager-687b7449bc-5hzqb"
Jul 08 16:40:19.878559 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:19.878500    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9t58\" (UniqueName: \"kubernetes.io/projected/b4e253de-313d-4de6-8e27-cc24d13b355c-kube-api-access-j9t58\") pod \"kserve-controller-manager-687b7449bc-5hzqb\" (UID: \"b4e253de-313d-4de6-8e27-cc24d13b355c\") " pod="kserve/kserve-controller-manager-687b7449bc-5hzqb"
Jul 08 16:40:19.878830 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:40:19.878641    2565 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 08 16:40:19.878830 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:40:19.878774    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e253de-313d-4de6-8e27-cc24d13b355c-cert podName:b4e253de-313d-4de6-8e27-cc24d13b355c nodeName:}" failed. No retries permitted until 2026-07-08 16:40:20.37874461 +0000 UTC m=+357.092106029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b4e253de-313d-4de6-8e27-cc24d13b355c-cert") pod "kserve-controller-manager-687b7449bc-5hzqb" (UID: "b4e253de-313d-4de6-8e27-cc24d13b355c") : secret "kserve-webhook-server-cert" not found
Jul 08 16:40:19.907447 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:19.907418    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9t58\" (UniqueName: \"kubernetes.io/projected/b4e253de-313d-4de6-8e27-cc24d13b355c-kube-api-access-j9t58\") pod \"kserve-controller-manager-687b7449bc-5hzqb\" (UID: \"b4e253de-313d-4de6-8e27-cc24d13b355c\") " pod="kserve/kserve-controller-manager-687b7449bc-5hzqb"
Jul 08 16:40:20.381720 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:20.381663    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4e253de-313d-4de6-8e27-cc24d13b355c-cert\") pod \"kserve-controller-manager-687b7449bc-5hzqb\" (UID: \"b4e253de-313d-4de6-8e27-cc24d13b355c\") " pod="kserve/kserve-controller-manager-687b7449bc-5hzqb"
Jul 08 16:40:20.384101 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:20.384076    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4e253de-313d-4de6-8e27-cc24d13b355c-cert\") pod \"kserve-controller-manager-687b7449bc-5hzqb\" (UID: \"b4e253de-313d-4de6-8e27-cc24d13b355c\") " pod="kserve/kserve-controller-manager-687b7449bc-5hzqb"
Jul 08 16:40:20.589893 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:20.589851    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-687b7449bc-5hzqb"
Jul 08 16:40:20.731183 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:20.731149    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-687b7449bc-5hzqb"]
Jul 08 16:40:20.734224 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:40:20.734188    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e253de_313d_4de6_8e27_cc24d13b355c.slice/crio-b4bc49e5cd51f8c3538063562afd188d747e4cbaaf31735373eb995f7ba8a0ab WatchSource:0}: Error finding container b4bc49e5cd51f8c3538063562afd188d747e4cbaaf31735373eb995f7ba8a0ab: Status 404 returned error can't find the container with id b4bc49e5cd51f8c3538063562afd188d747e4cbaaf31735373eb995f7ba8a0ab
Jul 08 16:40:20.735715 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:20.735696    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 16:40:21.476430 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:21.476395    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-687b7449bc-5hzqb" event={"ID":"b4e253de-313d-4de6-8e27-cc24d13b355c","Type":"ContainerStarted","Data":"b4bc49e5cd51f8c3538063562afd188d747e4cbaaf31735373eb995f7ba8a0ab"}
Jul 08 16:40:21.778272 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:21.778189    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:40:24.492124 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:24.492088    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-687b7449bc-5hzqb" event={"ID":"b4e253de-313d-4de6-8e27-cc24d13b355c","Type":"ContainerStarted","Data":"448934ec7eda4c2e7810a83f180d69c1f50cf84448f1337ffca8b1a4d487e4f7"}
Jul 08 16:40:24.492602 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:24.492188    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-687b7449bc-5hzqb"
Jul 08 16:40:24.516309 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:24.516263    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-687b7449bc-5hzqb" podStartSLOduration=2.55765865 podStartE2EDuration="5.516247168s" totalImagesPullingTime="2.958588518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:40:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:40:24.513744254 +0000 UTC m=+361.227105685" watchObservedRunningTime="2026-07-08 16:40:24.516247168 +0000 UTC m=+361.229608595"
Jul 08 16:40:55.501974 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:55.501944    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-687b7449bc-5hzqb"
Jul 08 16:40:57.170240 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:57.170199    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:40:57.270633 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:57.270596    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:40:57.451012 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:57.450982    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n"]
Jul 08 16:40:57.455896 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:57.455870    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n"
Jul 08 16:40:57.458851 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:57.458831    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mlhdn\""
Jul 08 16:40:57.458988 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:57.458960    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 08 16:40:57.459637 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:57.459620    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 08 16:40:57.467043 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:57.467017    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n"]
Jul 08 16:40:57.495169 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:57.495126    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fab922db-4404-48b3-936d-acfda0654393-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n\" (UID: \"fab922db-4404-48b3-936d-acfda0654393\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n"
Jul 08 16:40:57.495169 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:57.495175    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhzpl\" (UniqueName: \"kubernetes.io/projected/fab922db-4404-48b3-936d-acfda0654393-kube-api-access-rhzpl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n\" (UID: \"fab922db-4404-48b3-936d-acfda0654393\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n"
Jul 08 16:40:57.495507 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:57.495337    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fab922db-4404-48b3-936d-acfda0654393-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n\" (UID: \"fab922db-4404-48b3-936d-acfda0654393\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n"
Jul 08 16:40:57.596414 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:57.596375    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fab922db-4404-48b3-936d-acfda0654393-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n\" (UID: \"fab922db-4404-48b3-936d-acfda0654393\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n"
Jul 08 16:40:57.596607 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:57.596427    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fab922db-4404-48b3-936d-acfda0654393-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n\" (UID: \"fab922db-4404-48b3-936d-acfda0654393\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n"
Jul 08 16:40:57.596607 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:57.596451    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhzpl\" (UniqueName: \"kubernetes.io/projected/fab922db-4404-48b3-936d-acfda0654393-kube-api-access-rhzpl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n\" (UID: \"fab922db-4404-48b3-936d-acfda0654393\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n"
Jul 08 16:40:57.596870 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:57.596845    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fab922db-4404-48b3-936d-acfda0654393-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n\" (UID: \"fab922db-4404-48b3-936d-acfda0654393\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n"
Jul 08 16:40:57.596934 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:57.596864    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fab922db-4404-48b3-936d-acfda0654393-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n\" (UID: \"fab922db-4404-48b3-936d-acfda0654393\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n"
Jul 08 16:40:57.606914 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:57.606870    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhzpl\" (UniqueName: \"kubernetes.io/projected/fab922db-4404-48b3-936d-acfda0654393-kube-api-access-rhzpl\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n\" (UID: \"fab922db-4404-48b3-936d-acfda0654393\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n"
Jul 08 16:40:57.766154 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:57.766008    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n"
Jul 08 16:40:57.934878 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:57.934849    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n"]
Jul 08 16:40:57.937481 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:40:57.937429    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab922db_4404_48b3_936d_acfda0654393.slice/crio-b7bc773e427af81c44ebda61b171e3874b45496554692663fe544d32e2fb243b WatchSource:0}: Error finding container b7bc773e427af81c44ebda61b171e3874b45496554692663fe544d32e2fb243b: Status 404 returned error can't find the container with id b7bc773e427af81c44ebda61b171e3874b45496554692663fe544d32e2fb243b
Jul 08 16:40:58.640164 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:58.640131    2565 generic.go:356] "Generic (PLEG): container finished" podID="fab922db-4404-48b3-936d-acfda0654393" containerID="961b91ccc3a0e420b5c035fb19b488ef4a7126bf347dfd910216134d3408a1eb" exitCode=0
Jul 08 16:40:58.640531 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:58.640169    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n" event={"ID":"fab922db-4404-48b3-936d-acfda0654393","Type":"ContainerDied","Data":"961b91ccc3a0e420b5c035fb19b488ef4a7126bf347dfd910216134d3408a1eb"}
Jul 08 16:40:58.640531 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:58.640204    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n" event={"ID":"fab922db-4404-48b3-936d-acfda0654393","Type":"ContainerStarted","Data":"b7bc773e427af81c44ebda61b171e3874b45496554692663fe544d32e2fb243b"}
Jul 08 16:40:59.646521 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:59.646435    2565 generic.go:356] "Generic (PLEG): container finished" podID="fab922db-4404-48b3-936d-acfda0654393" containerID="947179eddacf9660b75753d992395646e728ef1a00e37805f66a2b1d4bd4118a" exitCode=0
Jul 08 16:40:59.646952 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:40:59.646526    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n" event={"ID":"fab922db-4404-48b3-936d-acfda0654393","Type":"ContainerDied","Data":"947179eddacf9660b75753d992395646e728ef1a00e37805f66a2b1d4bd4118a"}
Jul 08 16:41:00.653905 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:00.653871    2565 generic.go:356] "Generic (PLEG): container finished" podID="fab922db-4404-48b3-936d-acfda0654393" containerID="dc56643121fbcb559cb2427557f6adc6364903724d74497941184126e71fc14c" exitCode=0
Jul 08 16:41:00.654369 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:00.653926    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n" event={"ID":"fab922db-4404-48b3-936d-acfda0654393","Type":"ContainerDied","Data":"dc56643121fbcb559cb2427557f6adc6364903724d74497941184126e71fc14c"}
Jul 08 16:41:01.785625 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:01.785595    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n"
Jul 08 16:41:01.827836 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:01.827802    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fab922db-4404-48b3-936d-acfda0654393-util\") pod \"fab922db-4404-48b3-936d-acfda0654393\" (UID: \"fab922db-4404-48b3-936d-acfda0654393\") "
Jul 08 16:41:01.827998 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:01.827858    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhzpl\" (UniqueName: \"kubernetes.io/projected/fab922db-4404-48b3-936d-acfda0654393-kube-api-access-rhzpl\") pod \"fab922db-4404-48b3-936d-acfda0654393\" (UID: \"fab922db-4404-48b3-936d-acfda0654393\") "
Jul 08 16:41:01.827998 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:01.827886    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fab922db-4404-48b3-936d-acfda0654393-bundle\") pod \"fab922db-4404-48b3-936d-acfda0654393\" (UID: \"fab922db-4404-48b3-936d-acfda0654393\") "
Jul 08 16:41:01.828616 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:01.828590    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab922db-4404-48b3-936d-acfda0654393-bundle" (OuterVolumeSpecName: "bundle") pod "fab922db-4404-48b3-936d-acfda0654393" (UID: "fab922db-4404-48b3-936d-acfda0654393"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:41:01.829893 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:01.829872    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fab922db-4404-48b3-936d-acfda0654393-kube-api-access-rhzpl" (OuterVolumeSpecName: "kube-api-access-rhzpl") pod "fab922db-4404-48b3-936d-acfda0654393" (UID: "fab922db-4404-48b3-936d-acfda0654393"). InnerVolumeSpecName "kube-api-access-rhzpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:41:01.833615 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:01.833594    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fab922db-4404-48b3-936d-acfda0654393-util" (OuterVolumeSpecName: "util") pod "fab922db-4404-48b3-936d-acfda0654393" (UID: "fab922db-4404-48b3-936d-acfda0654393"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:41:01.928700 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:01.928644    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhzpl\" (UniqueName: \"kubernetes.io/projected/fab922db-4404-48b3-936d-acfda0654393-kube-api-access-rhzpl\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:41:01.928700 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:01.928697    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fab922db-4404-48b3-936d-acfda0654393-bundle\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:41:01.928882 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:01.928711    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fab922db-4404-48b3-936d-acfda0654393-util\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:41:02.664661 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:02.664628    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n"
Jul 08 16:41:02.664661 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:02.664644    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540tpw5n" event={"ID":"fab922db-4404-48b3-936d-acfda0654393","Type":"ContainerDied","Data":"b7bc773e427af81c44ebda61b171e3874b45496554692663fe544d32e2fb243b"}
Jul 08 16:41:02.664891 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:02.664710    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7bc773e427af81c44ebda61b171e3874b45496554692663fe544d32e2fb243b"
Jul 08 16:41:08.407881 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:08.407849    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:09.280925 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:09.280895    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:10.636480 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:10.636442    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f6rqh"]
Jul 08 16:41:10.636893 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:10.636763    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fab922db-4404-48b3-936d-acfda0654393" containerName="util"
Jul 08 16:41:10.636893 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:10.636776    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fab922db-4404-48b3-936d-acfda0654393" containerName="util"
Jul 08 16:41:10.636893 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:10.636788    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fab922db-4404-48b3-936d-acfda0654393" containerName="pull"
Jul 08 16:41:10.636893 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:10.636794    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fab922db-4404-48b3-936d-acfda0654393" containerName="pull"
Jul 08 16:41:10.636893 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:10.636803    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fab922db-4404-48b3-936d-acfda0654393" containerName="extract"
Jul 08 16:41:10.636893 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:10.636809    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fab922db-4404-48b3-936d-acfda0654393" containerName="extract"
Jul 08 16:41:10.636893 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:10.636863    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fab922db-4404-48b3-936d-acfda0654393" containerName="extract"
Jul 08 16:41:10.639684 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:10.639659    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f6rqh"
Jul 08 16:41:10.645802 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:10.645776    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 08 16:41:10.645943 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:10.645842    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-7xb62\""
Jul 08 16:41:10.646319 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:10.646052    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 08 16:41:10.661997 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:10.661968    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f6rqh"]
Jul 08 16:41:10.705667 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:10.705634    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f785bd80-a3c4-4104-bcf9-2671229019eb-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-f6rqh\" (UID: \"f785bd80-a3c4-4104-bcf9-2671229019eb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f6rqh"
Jul 08 16:41:10.705854 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:10.705670    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw76k\" (UniqueName: \"kubernetes.io/projected/f785bd80-a3c4-4104-bcf9-2671229019eb-kube-api-access-gw76k\") pod \"cert-manager-operator-controller-manager-684f79c9b9-f6rqh\" (UID: \"f785bd80-a3c4-4104-bcf9-2671229019eb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f6rqh"
Jul 08 16:41:10.806662 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:10.806626    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f785bd80-a3c4-4104-bcf9-2671229019eb-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-f6rqh\" (UID: \"f785bd80-a3c4-4104-bcf9-2671229019eb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f6rqh"
Jul 08 16:41:10.806662 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:10.806668    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw76k\" (UniqueName: \"kubernetes.io/projected/f785bd80-a3c4-4104-bcf9-2671229019eb-kube-api-access-gw76k\") pod \"cert-manager-operator-controller-manager-684f79c9b9-f6rqh\" (UID: \"f785bd80-a3c4-4104-bcf9-2671229019eb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f6rqh"
Jul 08 16:41:10.807073 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:10.807056    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f785bd80-a3c4-4104-bcf9-2671229019eb-tmp\") pod \"cert-manager-operator-controller-manager-684f79c9b9-f6rqh\" (UID: \"f785bd80-a3c4-4104-bcf9-2671229019eb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f6rqh"
Jul 08 16:41:10.821154 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:10.821124    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw76k\" (UniqueName: \"kubernetes.io/projected/f785bd80-a3c4-4104-bcf9-2671229019eb-kube-api-access-gw76k\") pod \"cert-manager-operator-controller-manager-684f79c9b9-f6rqh\" (UID: \"f785bd80-a3c4-4104-bcf9-2671229019eb\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f6rqh"
Jul 08 16:41:10.948649 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:10.948608    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f6rqh"
Jul 08 16:41:11.111618 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:11.111572    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f6rqh"]
Jul 08 16:41:11.115392 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:41:11.115357    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf785bd80_a3c4_4104_bcf9_2671229019eb.slice/crio-d6f4c8816c7f39d18d5cb177bda713ec04d6db7a7eeceef29fb95baf02078d78 WatchSource:0}: Error finding container d6f4c8816c7f39d18d5cb177bda713ec04d6db7a7eeceef29fb95baf02078d78: Status 404 returned error can't find the container with id d6f4c8816c7f39d18d5cb177bda713ec04d6db7a7eeceef29fb95baf02078d78
Jul 08 16:41:11.705140 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:11.705108    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f6rqh" event={"ID":"f785bd80-a3c4-4104-bcf9-2671229019eb","Type":"ContainerStarted","Data":"d6f4c8816c7f39d18d5cb177bda713ec04d6db7a7eeceef29fb95baf02078d78"}
Jul 08 16:41:12.400127 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:12.400091    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:12.744431 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:12.744389    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:13.717381 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:13.717338    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f6rqh" event={"ID":"f785bd80-a3c4-4104-bcf9-2671229019eb","Type":"ContainerStarted","Data":"5774ad0623404f212fc953f4d34546ae2d47da9fac7d193e1164588a5d1dae39"}
Jul 08 16:41:13.749473 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:13.749413    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-684f79c9b9-f6rqh" podStartSLOduration=1.798992842 podStartE2EDuration="3.749395996s" totalImagesPullingTime="1.950403154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:41:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:41:13.747506314 +0000 UTC m=+410.460867743" watchObservedRunningTime="2026-07-08 16:41:13.749395996 +0000 UTC m=+410.462757423"
Jul 08 16:41:14.777265 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:14.777220    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:14.878326 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:14.878292    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:17.419253 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:17.419210    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:17.494212 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:17.494181    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:19.614106 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:19.614071    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9"]
Jul 08 16:41:19.617538 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:19.617520    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9"
Jul 08 16:41:19.621108 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:19.621081    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 08 16:41:19.621232 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:19.621134    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mlhdn\""
Jul 08 16:41:19.621904 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:41:19.621880    2565 status_manager.go:1018] "Failed to get status for pod" err="pods \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9\" is forbidden: User \"system:node:ip-10-0-128-142.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-marketplace\": no relationship found between node 'ip-10-0-128-142.ec2.internal' and this object" podUID="7f46214e-c0a3-4628-aab3-ac49b228390d" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9"
Jul 08 16:41:19.622001 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:19.621907    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 08 16:41:19.629041 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:19.629021    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9"]
Jul 08 16:41:19.686831 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:19.686788    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f46214e-c0a3-4628-aab3-ac49b228390d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9\" (UID: \"7f46214e-c0a3-4628-aab3-ac49b228390d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9"
Jul 08 16:41:19.686991 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:19.686865    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f46214e-c0a3-4628-aab3-ac49b228390d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9\" (UID: \"7f46214e-c0a3-4628-aab3-ac49b228390d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9"
Jul 08 16:41:19.686991 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:19.686887    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6769\" (UniqueName: \"kubernetes.io/projected/7f46214e-c0a3-4628-aab3-ac49b228390d-kube-api-access-s6769\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9\" (UID: \"7f46214e-c0a3-4628-aab3-ac49b228390d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9"
Jul 08 16:41:19.788347 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:19.788306    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f46214e-c0a3-4628-aab3-ac49b228390d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9\" (UID: \"7f46214e-c0a3-4628-aab3-ac49b228390d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9"
Jul 08 16:41:19.788347 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:19.788344    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6769\" (UniqueName: \"kubernetes.io/projected/7f46214e-c0a3-4628-aab3-ac49b228390d-kube-api-access-s6769\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9\" (UID: \"7f46214e-c0a3-4628-aab3-ac49b228390d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9"
Jul 08 16:41:19.788582 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:19.788529    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f46214e-c0a3-4628-aab3-ac49b228390d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9\" (UID: \"7f46214e-c0a3-4628-aab3-ac49b228390d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9"
Jul 08 16:41:19.788762 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:19.788740    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f46214e-c0a3-4628-aab3-ac49b228390d-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9\" (UID: \"7f46214e-c0a3-4628-aab3-ac49b228390d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9"
Jul 08 16:41:19.788899 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:19.788878    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f46214e-c0a3-4628-aab3-ac49b228390d-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9\" (UID: \"7f46214e-c0a3-4628-aab3-ac49b228390d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9"
Jul 08 16:41:19.816202 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:19.816177    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6769\" (UniqueName: \"kubernetes.io/projected/7f46214e-c0a3-4628-aab3-ac49b228390d-kube-api-access-s6769\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9\" (UID: \"7f46214e-c0a3-4628-aab3-ac49b228390d\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9"
Jul 08 16:41:19.926872 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:19.926845    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9"
Jul 08 16:41:20.077829 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:20.077802    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9"]
Jul 08 16:41:20.080006 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:41:20.079970    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f46214e_c0a3_4628_aab3_ac49b228390d.slice/crio-5b86cff69925877e7cdca2153bfbda0a8e6b646ae969f80727b1129284a05d61 WatchSource:0}: Error finding container 5b86cff69925877e7cdca2153bfbda0a8e6b646ae969f80727b1129284a05d61: Status 404 returned error can't find the container with id 5b86cff69925877e7cdca2153bfbda0a8e6b646ae969f80727b1129284a05d61
Jul 08 16:41:20.184206 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:20.184167    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:20.263484 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:20.263451    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:20.338204 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:20.338158    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:20.750687 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:20.750644    2565 generic.go:356] "Generic (PLEG): container finished" podID="7f46214e-c0a3-4628-aab3-ac49b228390d" containerID="b47ea4c31ad86184af3f8972287f83e8fa6006fe1e81de57cb23c31b936fb241" exitCode=0
Jul 08 16:41:20.751088 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:20.750744    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9" event={"ID":"7f46214e-c0a3-4628-aab3-ac49b228390d","Type":"ContainerDied","Data":"b47ea4c31ad86184af3f8972287f83e8fa6006fe1e81de57cb23c31b936fb241"}
Jul 08 16:41:20.751088 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:20.750828    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9" event={"ID":"7f46214e-c0a3-4628-aab3-ac49b228390d","Type":"ContainerStarted","Data":"5b86cff69925877e7cdca2153bfbda0a8e6b646ae969f80727b1129284a05d61"}
Jul 08 16:41:22.132111 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:22.132056    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:22.766508 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:22.766454    2565 generic.go:356] "Generic (PLEG): container finished" podID="7f46214e-c0a3-4628-aab3-ac49b228390d" containerID="56e231cffbd2b33993dd101b31106b9977d3e5ce2a9a9c77ecf124e27c10df82" exitCode=0
Jul 08 16:41:22.766841 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:22.766546    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9" event={"ID":"7f46214e-c0a3-4628-aab3-ac49b228390d","Type":"ContainerDied","Data":"56e231cffbd2b33993dd101b31106b9977d3e5ce2a9a9c77ecf124e27c10df82"}
Jul 08 16:41:23.773259 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:23.773222    2565 generic.go:356] "Generic (PLEG): container finished" podID="7f46214e-c0a3-4628-aab3-ac49b228390d" containerID="6ac0c1e7e06e278dfddb66cabede283ee3180fb5f85881d0ba6a5d250175d32c" exitCode=0
Jul 08 16:41:23.773733 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:23.773304    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9" event={"ID":"7f46214e-c0a3-4628-aab3-ac49b228390d","Type":"ContainerDied","Data":"6ac0c1e7e06e278dfddb66cabede283ee3180fb5f85881d0ba6a5d250175d32c"}
Jul 08 16:41:24.612349 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:24.612317    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:24.708296 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:24.708251    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:24.910890 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:24.910862    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9"
Jul 08 16:41:25.040950 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:25.040900    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f46214e-c0a3-4628-aab3-ac49b228390d-bundle\") pod \"7f46214e-c0a3-4628-aab3-ac49b228390d\" (UID: \"7f46214e-c0a3-4628-aab3-ac49b228390d\") "
Jul 08 16:41:25.041261 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:25.040973    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6769\" (UniqueName: \"kubernetes.io/projected/7f46214e-c0a3-4628-aab3-ac49b228390d-kube-api-access-s6769\") pod \"7f46214e-c0a3-4628-aab3-ac49b228390d\" (UID: \"7f46214e-c0a3-4628-aab3-ac49b228390d\") "
Jul 08 16:41:25.041261 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:25.041038    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f46214e-c0a3-4628-aab3-ac49b228390d-util\") pod \"7f46214e-c0a3-4628-aab3-ac49b228390d\" (UID: \"7f46214e-c0a3-4628-aab3-ac49b228390d\") "
Jul 08 16:41:25.044232 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:25.041858    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f46214e-c0a3-4628-aab3-ac49b228390d-bundle" (OuterVolumeSpecName: "bundle") pod "7f46214e-c0a3-4628-aab3-ac49b228390d" (UID: "7f46214e-c0a3-4628-aab3-ac49b228390d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:41:25.050320 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:25.050179    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f46214e-c0a3-4628-aab3-ac49b228390d-util" (OuterVolumeSpecName: "util") pod "7f46214e-c0a3-4628-aab3-ac49b228390d" (UID: "7f46214e-c0a3-4628-aab3-ac49b228390d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:41:25.051278 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:25.051064    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f46214e-c0a3-4628-aab3-ac49b228390d-kube-api-access-s6769" (OuterVolumeSpecName: "kube-api-access-s6769") pod "7f46214e-c0a3-4628-aab3-ac49b228390d" (UID: "7f46214e-c0a3-4628-aab3-ac49b228390d"). InnerVolumeSpecName "kube-api-access-s6769". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:41:25.143414 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:25.143313    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7f46214e-c0a3-4628-aab3-ac49b228390d-bundle\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:41:25.143414 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:25.143356    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6769\" (UniqueName: \"kubernetes.io/projected/7f46214e-c0a3-4628-aab3-ac49b228390d-kube-api-access-s6769\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:41:25.143414 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:25.143367    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7f46214e-c0a3-4628-aab3-ac49b228390d-util\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:41:25.783938 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:25.783903    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9" event={"ID":"7f46214e-c0a3-4628-aab3-ac49b228390d","Type":"ContainerDied","Data":"5b86cff69925877e7cdca2153bfbda0a8e6b646ae969f80727b1129284a05d61"}
Jul 08 16:41:25.784148 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:25.783980    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b86cff69925877e7cdca2153bfbda0a8e6b646ae969f80727b1129284a05d61"
Jul 08 16:41:25.784148 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:25.783916    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87f4tnd9"
Jul 08 16:41:25.894107 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:25.894074    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:27.481743 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:27.481705    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dfh8t"]
Jul 08 16:41:27.482205 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:27.482025    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f46214e-c0a3-4628-aab3-ac49b228390d" containerName="pull"
Jul 08 16:41:27.482205 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:27.482037    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f46214e-c0a3-4628-aab3-ac49b228390d" containerName="pull"
Jul 08 16:41:27.482205 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:27.482054    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f46214e-c0a3-4628-aab3-ac49b228390d" containerName="util"
Jul 08 16:41:27.482205 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:27.482060    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f46214e-c0a3-4628-aab3-ac49b228390d" containerName="util"
Jul 08 16:41:27.482205 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:27.482066    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f46214e-c0a3-4628-aab3-ac49b228390d" containerName="extract"
Jul 08 16:41:27.482205 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:27.482071    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f46214e-c0a3-4628-aab3-ac49b228390d" containerName="extract"
Jul 08 16:41:27.482205 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:27.482126    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f46214e-c0a3-4628-aab3-ac49b228390d" containerName="extract"
Jul 08 16:41:27.484634 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:27.484612    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dfh8t"
Jul 08 16:41:27.488273 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:27.488250    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-hfb42\""
Jul 08 16:41:27.488384 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:27.488251    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 08 16:41:27.488384 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:27.488274    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 08 16:41:27.500984 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:27.500951    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dfh8t"]
Jul 08 16:41:27.566243 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:27.566208    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ad4c580-ebff-40d3-9669-1c3a92c6d3bb-bound-sa-token\") pod \"cert-manager-55b4d954d5-dfh8t\" (UID: \"7ad4c580-ebff-40d3-9669-1c3a92c6d3bb\") " pod="cert-manager/cert-manager-55b4d954d5-dfh8t"
Jul 08 16:41:27.566448 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:27.566249    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxnqn\" (UniqueName: \"kubernetes.io/projected/7ad4c580-ebff-40d3-9669-1c3a92c6d3bb-kube-api-access-bxnqn\") pod \"cert-manager-55b4d954d5-dfh8t\" (UID: \"7ad4c580-ebff-40d3-9669-1c3a92c6d3bb\") " pod="cert-manager/cert-manager-55b4d954d5-dfh8t"
Jul 08 16:41:27.667588 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:27.667559    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ad4c580-ebff-40d3-9669-1c3a92c6d3bb-bound-sa-token\") pod \"cert-manager-55b4d954d5-dfh8t\" (UID: \"7ad4c580-ebff-40d3-9669-1c3a92c6d3bb\") " pod="cert-manager/cert-manager-55b4d954d5-dfh8t"
Jul 08 16:41:27.667785 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:27.667597    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxnqn\" (UniqueName: \"kubernetes.io/projected/7ad4c580-ebff-40d3-9669-1c3a92c6d3bb-kube-api-access-bxnqn\") pod \"cert-manager-55b4d954d5-dfh8t\" (UID: \"7ad4c580-ebff-40d3-9669-1c3a92c6d3bb\") " pod="cert-manager/cert-manager-55b4d954d5-dfh8t"
Jul 08 16:41:27.679469 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:27.679432    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ad4c580-ebff-40d3-9669-1c3a92c6d3bb-bound-sa-token\") pod \"cert-manager-55b4d954d5-dfh8t\" (UID: \"7ad4c580-ebff-40d3-9669-1c3a92c6d3bb\") " pod="cert-manager/cert-manager-55b4d954d5-dfh8t"
Jul 08 16:41:27.681853 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:27.681827    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxnqn\" (UniqueName: \"kubernetes.io/projected/7ad4c580-ebff-40d3-9669-1c3a92c6d3bb-kube-api-access-bxnqn\") pod \"cert-manager-55b4d954d5-dfh8t\" (UID: \"7ad4c580-ebff-40d3-9669-1c3a92c6d3bb\") " pod="cert-manager/cert-manager-55b4d954d5-dfh8t"
Jul 08 16:41:27.794724 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:27.794619    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-dfh8t"
Jul 08 16:41:27.945037 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:27.945010    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-dfh8t"]
Jul 08 16:41:27.946711 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:41:27.946667    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ad4c580_ebff_40d3_9669_1c3a92c6d3bb.slice/crio-411611b445513bf6a83d8f301d6953f5cbe08a118c026be66263a833e0aab8a6 WatchSource:0}: Error finding container 411611b445513bf6a83d8f301d6953f5cbe08a118c026be66263a833e0aab8a6: Status 404 returned error can't find the container with id 411611b445513bf6a83d8f301d6953f5cbe08a118c026be66263a833e0aab8a6
Jul 08 16:41:28.800469 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:28.800397    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dfh8t" event={"ID":"7ad4c580-ebff-40d3-9669-1c3a92c6d3bb","Type":"ContainerStarted","Data":"411611b445513bf6a83d8f301d6953f5cbe08a118c026be66263a833e0aab8a6"}
Jul 08 16:41:30.323669 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:30.323632    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:30.400877 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:30.400835    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:30.811600 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:30.811566    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-dfh8t" event={"ID":"7ad4c580-ebff-40d3-9669-1c3a92c6d3bb","Type":"ContainerStarted","Data":"b6ae4df95bc8e9f1217bf2a57794177ec98968d5cc8056a01a7d9654155756e8"}
Jul 08 16:41:30.836587 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:30.836476    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-dfh8t" podStartSLOduration=1.217692415 podStartE2EDuration="3.836459094s" totalImagesPullingTime="2.618766679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:41:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:41:30.836356075 +0000 UTC m=+427.549717532" watchObservedRunningTime="2026-07-08 16:41:30.836459094 +0000 UTC m=+427.549820521"
Jul 08 16:41:37.229604 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:37.229548    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:42.223555 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.223518    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r"]
Jul 08 16:41:42.228527 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.228501    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r"
Jul 08 16:41:42.229092 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.229057    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:42.231258 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.231216    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 08 16:41:42.231382 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.231238    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 08 16:41:42.231382 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.231249    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 08 16:41:42.231382 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.231269    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 08 16:41:42.231583 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.231295    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 08 16:41:42.231583 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.231297    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-97qdz\""
Jul 08 16:41:42.231583 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.231572    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 08 16:41:42.244942 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.244915    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r"]
Jul 08 16:41:42.286608 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.286576    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg6sp\" (UniqueName: \"kubernetes.io/projected/63601443-3f06-49ac-ada3-05f45b6924d2-kube-api-access-dg6sp\") pod \"istiod-openshift-gateway-94bb8fbfd-5kc7r\" (UID: \"63601443-3f06-49ac-ada3-05f45b6924d2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r"
Jul 08 16:41:42.286804 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.286628    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/63601443-3f06-49ac-ada3-05f45b6924d2-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-5kc7r\" (UID: \"63601443-3f06-49ac-ada3-05f45b6924d2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r"
Jul 08 16:41:42.286804 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.286706    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63601443-3f06-49ac-ada3-05f45b6924d2-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-5kc7r\" (UID: \"63601443-3f06-49ac-ada3-05f45b6924d2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r"
Jul 08 16:41:42.286804 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.286740    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/63601443-3f06-49ac-ada3-05f45b6924d2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-5kc7r\" (UID: \"63601443-3f06-49ac-ada3-05f45b6924d2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r"
Jul 08 16:41:42.286804 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.286784    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/63601443-3f06-49ac-ada3-05f45b6924d2-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-5kc7r\" (UID: \"63601443-3f06-49ac-ada3-05f45b6924d2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r"
Jul 08 16:41:42.286804 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.286801    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/63601443-3f06-49ac-ada3-05f45b6924d2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-5kc7r\" (UID: \"63601443-3f06-49ac-ada3-05f45b6924d2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r"
Jul 08 16:41:42.286993 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.286831    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/63601443-3f06-49ac-ada3-05f45b6924d2-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-5kc7r\" (UID: \"63601443-3f06-49ac-ada3-05f45b6924d2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r"
Jul 08 16:41:42.292299 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.292274    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:42.387229 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.387194    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/63601443-3f06-49ac-ada3-05f45b6924d2-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-5kc7r\" (UID: \"63601443-3f06-49ac-ada3-05f45b6924d2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r"
Jul 08 16:41:42.387434 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.387239    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63601443-3f06-49ac-ada3-05f45b6924d2-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-5kc7r\" (UID: \"63601443-3f06-49ac-ada3-05f45b6924d2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r"
Jul 08 16:41:42.387434 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.387258    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/63601443-3f06-49ac-ada3-05f45b6924d2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-5kc7r\" (UID: \"63601443-3f06-49ac-ada3-05f45b6924d2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r"
Jul 08 16:41:42.387434 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.387280    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/63601443-3f06-49ac-ada3-05f45b6924d2-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-5kc7r\" (UID: \"63601443-3f06-49ac-ada3-05f45b6924d2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r"
Jul 08 16:41:42.387434 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.387296    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/63601443-3f06-49ac-ada3-05f45b6924d2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-5kc7r\" (UID: \"63601443-3f06-49ac-ada3-05f45b6924d2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r"
Jul 08 16:41:42.387434 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.387318    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/63601443-3f06-49ac-ada3-05f45b6924d2-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-5kc7r\" (UID: \"63601443-3f06-49ac-ada3-05f45b6924d2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r"
Jul 08 16:41:42.387434 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.387372    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg6sp\" (UniqueName: \"kubernetes.io/projected/63601443-3f06-49ac-ada3-05f45b6924d2-kube-api-access-dg6sp\") pod \"istiod-openshift-gateway-94bb8fbfd-5kc7r\" (UID: \"63601443-3f06-49ac-ada3-05f45b6924d2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r"
Jul 08 16:41:42.388082 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.388053    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/63601443-3f06-49ac-ada3-05f45b6924d2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-5kc7r\" (UID: \"63601443-3f06-49ac-ada3-05f45b6924d2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r"
Jul 08 16:41:42.389778 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.389744    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/63601443-3f06-49ac-ada3-05f45b6924d2-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-5kc7r\" (UID: \"63601443-3f06-49ac-ada3-05f45b6924d2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r"
Jul 08 16:41:42.389917 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.389899    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/63601443-3f06-49ac-ada3-05f45b6924d2-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-5kc7r\" (UID: \"63601443-3f06-49ac-ada3-05f45b6924d2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r"
Jul 08 16:41:42.390106 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.390086    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/63601443-3f06-49ac-ada3-05f45b6924d2-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-5kc7r\" (UID: \"63601443-3f06-49ac-ada3-05f45b6924d2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r"
Jul 08 16:41:42.390164 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.390119    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/63601443-3f06-49ac-ada3-05f45b6924d2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-5kc7r\" (UID: \"63601443-3f06-49ac-ada3-05f45b6924d2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r"
Jul 08 16:41:42.399999 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.399960    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63601443-3f06-49ac-ada3-05f45b6924d2-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-5kc7r\" (UID: \"63601443-3f06-49ac-ada3-05f45b6924d2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r"
Jul 08 16:41:42.400165 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.400043    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg6sp\" (UniqueName: \"kubernetes.io/projected/63601443-3f06-49ac-ada3-05f45b6924d2-kube-api-access-dg6sp\") pod \"istiod-openshift-gateway-94bb8fbfd-5kc7r\" (UID: \"63601443-3f06-49ac-ada3-05f45b6924d2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r"
Jul 08 16:41:42.538377 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.538264    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r"
Jul 08 16:41:42.709856 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.709827    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r"]
Jul 08 16:41:42.711288 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:41:42.711259    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63601443_3f06_49ac_ada3_05f45b6924d2.slice/crio-1499d24231d237e2b5332112e960abaf63c49fd0dc0b0867134c4d79b24c023b WatchSource:0}: Error finding container 1499d24231d237e2b5332112e960abaf63c49fd0dc0b0867134c4d79b24c023b: Status 404 returned error can't find the container with id 1499d24231d237e2b5332112e960abaf63c49fd0dc0b0867134c4d79b24c023b
Jul 08 16:41:42.877983 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:42.877885    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r" event={"ID":"63601443-3f06-49ac-ada3-05f45b6924d2","Type":"ContainerStarted","Data":"1499d24231d237e2b5332112e960abaf63c49fd0dc0b0867134c4d79b24c023b"}
Jul 08 16:41:43.557217 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:43.557181    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:46.311253 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:46.311216    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 08 16:41:46.311518 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:46.311296    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 08 16:41:46.901442 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:46.901398    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r" event={"ID":"63601443-3f06-49ac-ada3-05f45b6924d2","Type":"ContainerStarted","Data":"cd892c3f5ad857eaaffaea6f2c4651eb032b5c1593af85fb21244ce227ca3f3e"}
Jul 08 16:41:46.901644 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:46.901463    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r"
Jul 08 16:41:46.928119 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:46.928074    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r" podStartSLOduration=1.330495998 podStartE2EDuration="4.928059122s" totalImagesPullingTime="3.597563124s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:41:46.925736399 +0000 UTC m=+443.639097826" watchObservedRunningTime="2026-07-08 16:41:46.928059122 +0000 UTC m=+443.641420549"
Jul 08 16:41:47.309370 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:47.309319    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:47.397292 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:47.397256    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:47.907475 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:47.907429    2565 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-5kc7r container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 08 16:41:47.907638 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:47.907498    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r" podUID="63601443-3f06-49ac-ada3-05f45b6924d2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 08 16:41:50.907304 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:50.907271    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-5kc7r"
Jul 08 16:41:51.473170 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:51.473135    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:51.545193 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:51.545154    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:54.652352 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:54.652317    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:57.598609 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:57.598564    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:41:57.696123 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:41:57.696090    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:00.317164 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:00.317126    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:00.413617 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:00.413585    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:03.484477 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.484440    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7"]
Jul 08 16:42:03.489652 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.489627    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7"
Jul 08 16:42:03.492195 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.492166    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 08 16:42:03.492329 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.492217    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mlhdn\""
Jul 08 16:42:03.492794 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.492770    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 08 16:42:03.499935 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.499910    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7"]
Jul 08 16:42:03.568158 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.568125    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad1ecd44-907c-4d52-91b9-14442cce8a85-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7\" (UID: \"ad1ecd44-907c-4d52-91b9-14442cce8a85\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7"
Jul 08 16:42:03.568342 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.568204    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad1ecd44-907c-4d52-91b9-14442cce8a85-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7\" (UID: \"ad1ecd44-907c-4d52-91b9-14442cce8a85\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7"
Jul 08 16:42:03.568421 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.568229    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28jhw\" (UniqueName: \"kubernetes.io/projected/ad1ecd44-907c-4d52-91b9-14442cce8a85-kube-api-access-28jhw\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7\" (UID: \"ad1ecd44-907c-4d52-91b9-14442cce8a85\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7"
Jul 08 16:42:03.577355 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.577304    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj"]
Jul 08 16:42:03.580406 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.580388    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj"
Jul 08 16:42:03.589305 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.589280    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj"]
Jul 08 16:42:03.669304 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.669267    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad1ecd44-907c-4d52-91b9-14442cce8a85-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7\" (UID: \"ad1ecd44-907c-4d52-91b9-14442cce8a85\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7"
Jul 08 16:42:03.669510 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.669311    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28jhw\" (UniqueName: \"kubernetes.io/projected/ad1ecd44-907c-4d52-91b9-14442cce8a85-kube-api-access-28jhw\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7\" (UID: \"ad1ecd44-907c-4d52-91b9-14442cce8a85\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7"
Jul 08 16:42:03.669510 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.669363    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad1ecd44-907c-4d52-91b9-14442cce8a85-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7\" (UID: \"ad1ecd44-907c-4d52-91b9-14442cce8a85\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7"
Jul 08 16:42:03.669510 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.669393    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/827dd06e-8dc6-4598-a094-e9e7ae1d220d-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj\" (UID: \"827dd06e-8dc6-4598-a094-e9e7ae1d220d\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj"
Jul 08 16:42:03.669510 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.669429    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/827dd06e-8dc6-4598-a094-e9e7ae1d220d-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj\" (UID: \"827dd06e-8dc6-4598-a094-e9e7ae1d220d\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj"
Jul 08 16:42:03.669510 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.669461    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbvgb\" (UniqueName: \"kubernetes.io/projected/827dd06e-8dc6-4598-a094-e9e7ae1d220d-kube-api-access-dbvgb\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj\" (UID: \"827dd06e-8dc6-4598-a094-e9e7ae1d220d\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj"
Jul 08 16:42:03.669776 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.669747    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad1ecd44-907c-4d52-91b9-14442cce8a85-bundle\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7\" (UID: \"ad1ecd44-907c-4d52-91b9-14442cce8a85\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7"
Jul 08 16:42:03.669818 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.669758    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad1ecd44-907c-4d52-91b9-14442cce8a85-util\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7\" (UID: \"ad1ecd44-907c-4d52-91b9-14442cce8a85\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7"
Jul 08 16:42:03.673262 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.673237    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4"]
Jul 08 16:42:03.676326 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.676311    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4"
Jul 08 16:42:03.682156 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.682132    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28jhw\" (UniqueName: \"kubernetes.io/projected/ad1ecd44-907c-4d52-91b9-14442cce8a85-kube-api-access-28jhw\") pod \"0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7\" (UID: \"ad1ecd44-907c-4d52-91b9-14442cce8a85\") " pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7"
Jul 08 16:42:03.686747 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.686723    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4"]
Jul 08 16:42:03.760382 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.760298    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6"]
Jul 08 16:42:03.763479 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.763463    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6"
Jul 08 16:42:03.770004 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.769979    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb410cb9-f79c-49a3-9e8e-cfc988360df7-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4\" (UID: \"fb410cb9-f79c-49a3-9e8e-cfc988360df7\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4"
Jul 08 16:42:03.770097 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.770025    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/827dd06e-8dc6-4598-a094-e9e7ae1d220d-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj\" (UID: \"827dd06e-8dc6-4598-a094-e9e7ae1d220d\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj"
Jul 08 16:42:03.770097 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.770047    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbvgb\" (UniqueName: \"kubernetes.io/projected/827dd06e-8dc6-4598-a094-e9e7ae1d220d-kube-api-access-dbvgb\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj\" (UID: \"827dd06e-8dc6-4598-a094-e9e7ae1d220d\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj"
Jul 08 16:42:03.770097 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.770093    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb410cb9-f79c-49a3-9e8e-cfc988360df7-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4\" (UID: \"fb410cb9-f79c-49a3-9e8e-cfc988360df7\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4"
Jul 08 16:42:03.770198 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.770128    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npd6v\" (UniqueName: \"kubernetes.io/projected/fb410cb9-f79c-49a3-9e8e-cfc988360df7-kube-api-access-npd6v\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4\" (UID: \"fb410cb9-f79c-49a3-9e8e-cfc988360df7\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4"
Jul 08 16:42:03.770198 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.770168    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/827dd06e-8dc6-4598-a094-e9e7ae1d220d-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj\" (UID: \"827dd06e-8dc6-4598-a094-e9e7ae1d220d\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj"
Jul 08 16:42:03.770511 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.770492    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/827dd06e-8dc6-4598-a094-e9e7ae1d220d-util\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj\" (UID: \"827dd06e-8dc6-4598-a094-e9e7ae1d220d\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj"
Jul 08 16:42:03.770575 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.770540    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/827dd06e-8dc6-4598-a094-e9e7ae1d220d-bundle\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj\" (UID: \"827dd06e-8dc6-4598-a094-e9e7ae1d220d\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj"
Jul 08 16:42:03.773619 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.773592    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6"]
Jul 08 16:42:03.778968 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.778939    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbvgb\" (UniqueName: \"kubernetes.io/projected/827dd06e-8dc6-4598-a094-e9e7ae1d220d-kube-api-access-dbvgb\") pod \"af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj\" (UID: \"827dd06e-8dc6-4598-a094-e9e7ae1d220d\") " pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj"
Jul 08 16:42:03.800470 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.800439    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7"
Jul 08 16:42:03.871432 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.871393    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npd6v\" (UniqueName: \"kubernetes.io/projected/fb410cb9-f79c-49a3-9e8e-cfc988360df7-kube-api-access-npd6v\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4\" (UID: \"fb410cb9-f79c-49a3-9e8e-cfc988360df7\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4"
Jul 08 16:42:03.871612 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.871443    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f2796a7-8378-4dc7-84f0-59ef3a6e9666-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6\" (UID: \"8f2796a7-8378-4dc7-84f0-59ef3a6e9666\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6"
Jul 08 16:42:03.871612 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.871477    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb410cb9-f79c-49a3-9e8e-cfc988360df7-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4\" (UID: \"fb410cb9-f79c-49a3-9e8e-cfc988360df7\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4"
Jul 08 16:42:03.871612 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.871524    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzpvt\" (UniqueName: \"kubernetes.io/projected/8f2796a7-8378-4dc7-84f0-59ef3a6e9666-kube-api-access-vzpvt\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6\" (UID: \"8f2796a7-8378-4dc7-84f0-59ef3a6e9666\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6"
Jul 08 16:42:03.871800 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.871722    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f2796a7-8378-4dc7-84f0-59ef3a6e9666-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6\" (UID: \"8f2796a7-8378-4dc7-84f0-59ef3a6e9666\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6"
Jul 08 16:42:03.871870 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.871848    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb410cb9-f79c-49a3-9e8e-cfc988360df7-util\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4\" (UID: \"fb410cb9-f79c-49a3-9e8e-cfc988360df7\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4"
Jul 08 16:42:03.871870 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.871857    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb410cb9-f79c-49a3-9e8e-cfc988360df7-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4\" (UID: \"fb410cb9-f79c-49a3-9e8e-cfc988360df7\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4"
Jul 08 16:42:03.872324 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.872301    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb410cb9-f79c-49a3-9e8e-cfc988360df7-bundle\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4\" (UID: \"fb410cb9-f79c-49a3-9e8e-cfc988360df7\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4"
Jul 08 16:42:03.882224 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.882199    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npd6v\" (UniqueName: \"kubernetes.io/projected/fb410cb9-f79c-49a3-9e8e-cfc988360df7-kube-api-access-npd6v\") pod \"f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4\" (UID: \"fb410cb9-f79c-49a3-9e8e-cfc988360df7\") " pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4"
Jul 08 16:42:03.891430 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.891105    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj"
Jul 08 16:42:03.947006 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.946971    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7"]
Jul 08 16:42:03.949961 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:42:03.949828    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad1ecd44_907c_4d52_91b9_14442cce8a85.slice/crio-184da0da5c2d491a5957606de59da485a8147c0c4c891a340fd4e62a4b756b67 WatchSource:0}: Error finding container 184da0da5c2d491a5957606de59da485a8147c0c4c891a340fd4e62a4b756b67: Status 404 returned error can't find the container with id 184da0da5c2d491a5957606de59da485a8147c0c4c891a340fd4e62a4b756b67
Jul 08 16:42:03.973110 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.973085    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f2796a7-8378-4dc7-84f0-59ef3a6e9666-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6\" (UID: \"8f2796a7-8378-4dc7-84f0-59ef3a6e9666\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6"
Jul 08 16:42:03.973233 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.973155    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f2796a7-8378-4dc7-84f0-59ef3a6e9666-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6\" (UID: \"8f2796a7-8378-4dc7-84f0-59ef3a6e9666\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6"
Jul 08 16:42:03.973233 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.973203    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzpvt\" (UniqueName: \"kubernetes.io/projected/8f2796a7-8378-4dc7-84f0-59ef3a6e9666-kube-api-access-vzpvt\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6\" (UID: \"8f2796a7-8378-4dc7-84f0-59ef3a6e9666\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6"
Jul 08 16:42:03.973477 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.973456    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f2796a7-8378-4dc7-84f0-59ef3a6e9666-util\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6\" (UID: \"8f2796a7-8378-4dc7-84f0-59ef3a6e9666\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6"
Jul 08 16:42:03.973551 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.973527    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f2796a7-8378-4dc7-84f0-59ef3a6e9666-bundle\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6\" (UID: \"8f2796a7-8378-4dc7-84f0-59ef3a6e9666\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6"
Jul 08 16:42:03.984294 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.984255    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7" event={"ID":"ad1ecd44-907c-4d52-91b9-14442cce8a85","Type":"ContainerStarted","Data":"184da0da5c2d491a5957606de59da485a8147c0c4c891a340fd4e62a4b756b67"}
Jul 08 16:42:03.985126 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.985098    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzpvt\" (UniqueName: \"kubernetes.io/projected/8f2796a7-8378-4dc7-84f0-59ef3a6e9666-kube-api-access-vzpvt\") pod \"299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6\" (UID: \"8f2796a7-8378-4dc7-84f0-59ef3a6e9666\") " pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6"
Jul 08 16:42:03.993228 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:03.993203    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4"
Jul 08 16:42:04.036847 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:04.036765    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj"]
Jul 08 16:42:04.040781 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:42:04.040753    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod827dd06e_8dc6_4598_a094_e9e7ae1d220d.slice/crio-11c1127d8294c65d0c4de37170f5dace673ed8295bfe33573c43ac2bf27cebc7 WatchSource:0}: Error finding container 11c1127d8294c65d0c4de37170f5dace673ed8295bfe33573c43ac2bf27cebc7: Status 404 returned error can't find the container with id 11c1127d8294c65d0c4de37170f5dace673ed8295bfe33573c43ac2bf27cebc7
Jul 08 16:42:04.072538 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:04.072508    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6"
Jul 08 16:42:04.145722 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:04.145697    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4"]
Jul 08 16:42:04.148905 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:42:04.148864    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb410cb9_f79c_49a3_9e8e_cfc988360df7.slice/crio-89f477af01cef91206efa40994bdd8367ed09b11ffb2e467e95cef93cafadc35 WatchSource:0}: Error finding container 89f477af01cef91206efa40994bdd8367ed09b11ffb2e467e95cef93cafadc35: Status 404 returned error can't find the container with id 89f477af01cef91206efa40994bdd8367ed09b11ffb2e467e95cef93cafadc35
Jul 08 16:42:04.229296 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:04.229274    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6"]
Jul 08 16:42:04.258068 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:42:04.258035    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f2796a7_8378_4dc7_84f0_59ef3a6e9666.slice/crio-6e78357be9d71557d333dd30f4094c662bc19b1193846c37f4dad585ecae7b5a WatchSource:0}: Error finding container 6e78357be9d71557d333dd30f4094c662bc19b1193846c37f4dad585ecae7b5a: Status 404 returned error can't find the container with id 6e78357be9d71557d333dd30f4094c662bc19b1193846c37f4dad585ecae7b5a
Jul 08 16:42:04.990324 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:04.990275    2565 generic.go:356] "Generic (PLEG): container finished" podID="ad1ecd44-907c-4d52-91b9-14442cce8a85" containerID="b0d04b76b3448a1fc19ea66050a239d38a126b33e50ca08d1819e1ca91a76c60" exitCode=0
Jul 08 16:42:04.990808 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:04.990365    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7" event={"ID":"ad1ecd44-907c-4d52-91b9-14442cce8a85","Type":"ContainerDied","Data":"b0d04b76b3448a1fc19ea66050a239d38a126b33e50ca08d1819e1ca91a76c60"}
Jul 08 16:42:04.992005 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:04.991882    2565 generic.go:356] "Generic (PLEG): container finished" podID="8f2796a7-8378-4dc7-84f0-59ef3a6e9666" containerID="3422fc5ccb841d26aadda60629f3332b3e46d12d4501af2b8e171d6b075f9421" exitCode=0
Jul 08 16:42:04.992070 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:04.991995    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6" event={"ID":"8f2796a7-8378-4dc7-84f0-59ef3a6e9666","Type":"ContainerDied","Data":"3422fc5ccb841d26aadda60629f3332b3e46d12d4501af2b8e171d6b075f9421"}
Jul 08 16:42:04.992070 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:04.992045    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6" event={"ID":"8f2796a7-8378-4dc7-84f0-59ef3a6e9666","Type":"ContainerStarted","Data":"6e78357be9d71557d333dd30f4094c662bc19b1193846c37f4dad585ecae7b5a"}
Jul 08 16:42:04.993502 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:04.993482    2565 generic.go:356] "Generic (PLEG): container finished" podID="fb410cb9-f79c-49a3-9e8e-cfc988360df7" containerID="c3c2151bd1fac2e8f9b21af3247a74d4a106ed251bd15553547619f9f8e2c290" exitCode=0
Jul 08 16:42:04.993574 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:04.993558    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4" event={"ID":"fb410cb9-f79c-49a3-9e8e-cfc988360df7","Type":"ContainerDied","Data":"c3c2151bd1fac2e8f9b21af3247a74d4a106ed251bd15553547619f9f8e2c290"}
Jul 08 16:42:04.993626 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:04.993597    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4" event={"ID":"fb410cb9-f79c-49a3-9e8e-cfc988360df7","Type":"ContainerStarted","Data":"89f477af01cef91206efa40994bdd8367ed09b11ffb2e467e95cef93cafadc35"}
Jul 08 16:42:04.995120 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:04.995094    2565 generic.go:356] "Generic (PLEG): container finished" podID="827dd06e-8dc6-4598-a094-e9e7ae1d220d" containerID="8878b5c162dc736f1cc46aaa8a1a3222b6e9d1f05765ea40116b828020199f8f" exitCode=0
Jul 08 16:42:04.995219 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:04.995131    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj" event={"ID":"827dd06e-8dc6-4598-a094-e9e7ae1d220d","Type":"ContainerDied","Data":"8878b5c162dc736f1cc46aaa8a1a3222b6e9d1f05765ea40116b828020199f8f"}
Jul 08 16:42:04.995219 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:04.995167    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj" event={"ID":"827dd06e-8dc6-4598-a094-e9e7ae1d220d","Type":"ContainerStarted","Data":"11c1127d8294c65d0c4de37170f5dace673ed8295bfe33573c43ac2bf27cebc7"}
Jul 08 16:42:05.773365 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:05.773315    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:06.003225 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:06.003190    2565 generic.go:356] "Generic (PLEG): container finished" podID="ad1ecd44-907c-4d52-91b9-14442cce8a85" containerID="3faa39e8b9d489d317ec3b7f68fc217c24842133c903fe5c14a68880fad39e21" exitCode=0
Jul 08 16:42:06.003585 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:06.003292    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7" event={"ID":"ad1ecd44-907c-4d52-91b9-14442cce8a85","Type":"ContainerDied","Data":"3faa39e8b9d489d317ec3b7f68fc217c24842133c903fe5c14a68880fad39e21"}
Jul 08 16:42:06.006089 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:06.006066    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6" event={"ID":"8f2796a7-8378-4dc7-84f0-59ef3a6e9666","Type":"ContainerStarted","Data":"2aa6ab7cdabfb66b5b2e159708b5cd3bdd26ccea639a90c09c1318ee41a9eeaa"}
Jul 08 16:42:06.008229 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:06.008202    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4" event={"ID":"fb410cb9-f79c-49a3-9e8e-cfc988360df7","Type":"ContainerStarted","Data":"a9300c0ff65140363b65d653f006ed855f99125cc20e3a52fb96b7c700867b49"}
Jul 08 16:42:06.010317 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:06.010126    2565 generic.go:356] "Generic (PLEG): container finished" podID="827dd06e-8dc6-4598-a094-e9e7ae1d220d" containerID="39ef5a3e14cf6f3672c3a3ec0998b4a203c4251486ac374f555cf1a814b80f92" exitCode=0
Jul 08 16:42:06.010317 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:06.010206    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj" event={"ID":"827dd06e-8dc6-4598-a094-e9e7ae1d220d","Type":"ContainerDied","Data":"39ef5a3e14cf6f3672c3a3ec0998b4a203c4251486ac374f555cf1a814b80f92"}
Jul 08 16:42:07.023481 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:07.023442    2565 generic.go:356] "Generic (PLEG): container finished" podID="ad1ecd44-907c-4d52-91b9-14442cce8a85" containerID="65693f8168b91e015bccb5cd9f72b7818c581e96b41d16234ebab8149ece61da" exitCode=0
Jul 08 16:42:07.023957 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:07.023529    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7" event={"ID":"ad1ecd44-907c-4d52-91b9-14442cce8a85","Type":"ContainerDied","Data":"65693f8168b91e015bccb5cd9f72b7818c581e96b41d16234ebab8149ece61da"}
Jul 08 16:42:07.025258 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:07.025231    2565 generic.go:356] "Generic (PLEG): container finished" podID="8f2796a7-8378-4dc7-84f0-59ef3a6e9666" containerID="2aa6ab7cdabfb66b5b2e159708b5cd3bdd26ccea639a90c09c1318ee41a9eeaa" exitCode=0
Jul 08 16:42:07.025349 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:07.025309    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6" event={"ID":"8f2796a7-8378-4dc7-84f0-59ef3a6e9666","Type":"ContainerDied","Data":"2aa6ab7cdabfb66b5b2e159708b5cd3bdd26ccea639a90c09c1318ee41a9eeaa"}
Jul 08 16:42:07.027164 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:07.027142    2565 generic.go:356] "Generic (PLEG): container finished" podID="fb410cb9-f79c-49a3-9e8e-cfc988360df7" containerID="a9300c0ff65140363b65d653f006ed855f99125cc20e3a52fb96b7c700867b49" exitCode=0
Jul 08 16:42:07.027281 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:07.027207    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4" event={"ID":"fb410cb9-f79c-49a3-9e8e-cfc988360df7","Type":"ContainerDied","Data":"a9300c0ff65140363b65d653f006ed855f99125cc20e3a52fb96b7c700867b49"}
Jul 08 16:42:07.029449 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:07.029428    2565 generic.go:356] "Generic (PLEG): container finished" podID="827dd06e-8dc6-4598-a094-e9e7ae1d220d" containerID="39eaf566f92a07a30dca14f201e9cd140a34d89c507ddc427c787d0a54de9a39" exitCode=0
Jul 08 16:42:07.029533 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:07.029466    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj" event={"ID":"827dd06e-8dc6-4598-a094-e9e7ae1d220d","Type":"ContainerDied","Data":"39eaf566f92a07a30dca14f201e9cd140a34d89c507ddc427c787d0a54de9a39"}
Jul 08 16:42:08.036059 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:08.036020    2565 generic.go:356] "Generic (PLEG): container finished" podID="fb410cb9-f79c-49a3-9e8e-cfc988360df7" containerID="879adcbf44d71f2984f7b4f4eefcb9acc1ceaa50b7019583266694ab18e7ae20" exitCode=0
Jul 08 16:42:08.036508 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:08.036111    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4" event={"ID":"fb410cb9-f79c-49a3-9e8e-cfc988360df7","Type":"ContainerDied","Data":"879adcbf44d71f2984f7b4f4eefcb9acc1ceaa50b7019583266694ab18e7ae20"}
Jul 08 16:42:08.038219 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:08.038192    2565 generic.go:356] "Generic (PLEG): container finished" podID="8f2796a7-8378-4dc7-84f0-59ef3a6e9666" containerID="7176e4049389c1374c0c14062a53bd0ef65600a9e06a67ae65309e820b87a8a0" exitCode=0
Jul 08 16:42:08.038364 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:08.038272    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6" event={"ID":"8f2796a7-8378-4dc7-84f0-59ef3a6e9666","Type":"ContainerDied","Data":"7176e4049389c1374c0c14062a53bd0ef65600a9e06a67ae65309e820b87a8a0"}
Jul 08 16:42:08.205192 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:08.205163    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7"
Jul 08 16:42:08.208231 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:08.208209    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj"
Jul 08 16:42:08.315632 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:08.315540    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28jhw\" (UniqueName: \"kubernetes.io/projected/ad1ecd44-907c-4d52-91b9-14442cce8a85-kube-api-access-28jhw\") pod \"ad1ecd44-907c-4d52-91b9-14442cce8a85\" (UID: \"ad1ecd44-907c-4d52-91b9-14442cce8a85\") "
Jul 08 16:42:08.315632 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:08.315584    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad1ecd44-907c-4d52-91b9-14442cce8a85-util\") pod \"ad1ecd44-907c-4d52-91b9-14442cce8a85\" (UID: \"ad1ecd44-907c-4d52-91b9-14442cce8a85\") "
Jul 08 16:42:08.315632 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:08.315610    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad1ecd44-907c-4d52-91b9-14442cce8a85-bundle\") pod \"ad1ecd44-907c-4d52-91b9-14442cce8a85\" (UID: \"ad1ecd44-907c-4d52-91b9-14442cce8a85\") "
Jul 08 16:42:08.315924 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:08.315726    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/827dd06e-8dc6-4598-a094-e9e7ae1d220d-util\") pod \"827dd06e-8dc6-4598-a094-e9e7ae1d220d\" (UID: \"827dd06e-8dc6-4598-a094-e9e7ae1d220d\") "
Jul 08 16:42:08.315924 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:08.315746    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbvgb\" (UniqueName: \"kubernetes.io/projected/827dd06e-8dc6-4598-a094-e9e7ae1d220d-kube-api-access-dbvgb\") pod \"827dd06e-8dc6-4598-a094-e9e7ae1d220d\" (UID: \"827dd06e-8dc6-4598-a094-e9e7ae1d220d\") "
Jul 08 16:42:08.315924 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:08.315776    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/827dd06e-8dc6-4598-a094-e9e7ae1d220d-bundle\") pod \"827dd06e-8dc6-4598-a094-e9e7ae1d220d\" (UID: \"827dd06e-8dc6-4598-a094-e9e7ae1d220d\") "
Jul 08 16:42:08.316315 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:08.316288    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad1ecd44-907c-4d52-91b9-14442cce8a85-bundle" (OuterVolumeSpecName: "bundle") pod "ad1ecd44-907c-4d52-91b9-14442cce8a85" (UID: "ad1ecd44-907c-4d52-91b9-14442cce8a85"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:42:08.316443 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:08.316413    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827dd06e-8dc6-4598-a094-e9e7ae1d220d-bundle" (OuterVolumeSpecName: "bundle") pod "827dd06e-8dc6-4598-a094-e9e7ae1d220d" (UID: "827dd06e-8dc6-4598-a094-e9e7ae1d220d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:42:08.317909 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:08.317882    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad1ecd44-907c-4d52-91b9-14442cce8a85-kube-api-access-28jhw" (OuterVolumeSpecName: "kube-api-access-28jhw") pod "ad1ecd44-907c-4d52-91b9-14442cce8a85" (UID: "ad1ecd44-907c-4d52-91b9-14442cce8a85"). InnerVolumeSpecName "kube-api-access-28jhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:42:08.317999 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:08.317907    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/827dd06e-8dc6-4598-a094-e9e7ae1d220d-kube-api-access-dbvgb" (OuterVolumeSpecName: "kube-api-access-dbvgb") pod "827dd06e-8dc6-4598-a094-e9e7ae1d220d" (UID: "827dd06e-8dc6-4598-a094-e9e7ae1d220d"). InnerVolumeSpecName "kube-api-access-dbvgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:42:08.321431 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:08.321406    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad1ecd44-907c-4d52-91b9-14442cce8a85-util" (OuterVolumeSpecName: "util") pod "ad1ecd44-907c-4d52-91b9-14442cce8a85" (UID: "ad1ecd44-907c-4d52-91b9-14442cce8a85"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:42:08.321738 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:08.321718    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/827dd06e-8dc6-4598-a094-e9e7ae1d220d-util" (OuterVolumeSpecName: "util") pod "827dd06e-8dc6-4598-a094-e9e7ae1d220d" (UID: "827dd06e-8dc6-4598-a094-e9e7ae1d220d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:42:08.416508 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:08.416463    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/827dd06e-8dc6-4598-a094-e9e7ae1d220d-bundle\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:42:08.416508 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:08.416501    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28jhw\" (UniqueName: \"kubernetes.io/projected/ad1ecd44-907c-4d52-91b9-14442cce8a85-kube-api-access-28jhw\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:42:08.416508 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:08.416515    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ad1ecd44-907c-4d52-91b9-14442cce8a85-util\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:42:08.416791 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:08.416526    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ad1ecd44-907c-4d52-91b9-14442cce8a85-bundle\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:42:08.416791 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:08.416538    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/827dd06e-8dc6-4598-a094-e9e7ae1d220d-util\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:42:08.416791 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:08.416548    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbvgb\" (UniqueName: \"kubernetes.io/projected/827dd06e-8dc6-4598-a094-e9e7ae1d220d-kube-api-access-dbvgb\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:42:09.045346 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:09.045312    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7"
Jul 08 16:42:09.045346 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:09.045334    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/0e9e3357d40614682d1ebdb402e8ba62e1e258fb473e6f54140aafff36qb5x7" event={"ID":"ad1ecd44-907c-4d52-91b9-14442cce8a85","Type":"ContainerDied","Data":"184da0da5c2d491a5957606de59da485a8147c0c4c891a340fd4e62a4b756b67"}
Jul 08 16:42:09.045850 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:09.045390    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="184da0da5c2d491a5957606de59da485a8147c0c4c891a340fd4e62a4b756b67"
Jul 08 16:42:09.047335 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:09.047313    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj"
Jul 08 16:42:09.047438 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:09.047342    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/af2a5cb5935dca7d5808b44f590e0d3d1363f94abbf5c1d8a71e95fcabfrzcj" event={"ID":"827dd06e-8dc6-4598-a094-e9e7ae1d220d","Type":"ContainerDied","Data":"11c1127d8294c65d0c4de37170f5dace673ed8295bfe33573c43ac2bf27cebc7"}
Jul 08 16:42:09.047438 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:09.047388    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11c1127d8294c65d0c4de37170f5dace673ed8295bfe33573c43ac2bf27cebc7"
Jul 08 16:42:09.212207 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:09.212181    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6"
Jul 08 16:42:09.214132 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:09.214112    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4"
Jul 08 16:42:09.326222 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:09.326128    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npd6v\" (UniqueName: \"kubernetes.io/projected/fb410cb9-f79c-49a3-9e8e-cfc988360df7-kube-api-access-npd6v\") pod \"fb410cb9-f79c-49a3-9e8e-cfc988360df7\" (UID: \"fb410cb9-f79c-49a3-9e8e-cfc988360df7\") "
Jul 08 16:42:09.326222 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:09.326209    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzpvt\" (UniqueName: \"kubernetes.io/projected/8f2796a7-8378-4dc7-84f0-59ef3a6e9666-kube-api-access-vzpvt\") pod \"8f2796a7-8378-4dc7-84f0-59ef3a6e9666\" (UID: \"8f2796a7-8378-4dc7-84f0-59ef3a6e9666\") "
Jul 08 16:42:09.326460 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:09.326232    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb410cb9-f79c-49a3-9e8e-cfc988360df7-util\") pod \"fb410cb9-f79c-49a3-9e8e-cfc988360df7\" (UID: \"fb410cb9-f79c-49a3-9e8e-cfc988360df7\") "
Jul 08 16:42:09.326460 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:09.326272    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f2796a7-8378-4dc7-84f0-59ef3a6e9666-util\") pod \"8f2796a7-8378-4dc7-84f0-59ef3a6e9666\" (UID: \"8f2796a7-8378-4dc7-84f0-59ef3a6e9666\") "
Jul 08 16:42:09.326460 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:09.326339    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f2796a7-8378-4dc7-84f0-59ef3a6e9666-bundle\") pod \"8f2796a7-8378-4dc7-84f0-59ef3a6e9666\" (UID: \"8f2796a7-8378-4dc7-84f0-59ef3a6e9666\") "
Jul 08 16:42:09.326460 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:09.326387    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb410cb9-f79c-49a3-9e8e-cfc988360df7-bundle\") pod \"fb410cb9-f79c-49a3-9e8e-cfc988360df7\" (UID: \"fb410cb9-f79c-49a3-9e8e-cfc988360df7\") "
Jul 08 16:42:09.326873 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:09.326846    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2796a7-8378-4dc7-84f0-59ef3a6e9666-bundle" (OuterVolumeSpecName: "bundle") pod "8f2796a7-8378-4dc7-84f0-59ef3a6e9666" (UID: "8f2796a7-8378-4dc7-84f0-59ef3a6e9666"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:42:09.327290 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:09.327258    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb410cb9-f79c-49a3-9e8e-cfc988360df7-bundle" (OuterVolumeSpecName: "bundle") pod "fb410cb9-f79c-49a3-9e8e-cfc988360df7" (UID: "fb410cb9-f79c-49a3-9e8e-cfc988360df7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:42:09.328538 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:09.328506    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb410cb9-f79c-49a3-9e8e-cfc988360df7-kube-api-access-npd6v" (OuterVolumeSpecName: "kube-api-access-npd6v") pod "fb410cb9-f79c-49a3-9e8e-cfc988360df7" (UID: "fb410cb9-f79c-49a3-9e8e-cfc988360df7"). InnerVolumeSpecName "kube-api-access-npd6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:42:09.328825 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:09.328805    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f2796a7-8378-4dc7-84f0-59ef3a6e9666-kube-api-access-vzpvt" (OuterVolumeSpecName: "kube-api-access-vzpvt") pod "8f2796a7-8378-4dc7-84f0-59ef3a6e9666" (UID: "8f2796a7-8378-4dc7-84f0-59ef3a6e9666"). InnerVolumeSpecName "kube-api-access-vzpvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:42:09.332415 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:09.332397    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2796a7-8378-4dc7-84f0-59ef3a6e9666-util" (OuterVolumeSpecName: "util") pod "8f2796a7-8378-4dc7-84f0-59ef3a6e9666" (UID: "8f2796a7-8378-4dc7-84f0-59ef3a6e9666"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:42:09.333076 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:09.332976    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb410cb9-f79c-49a3-9e8e-cfc988360df7-util" (OuterVolumeSpecName: "util") pod "fb410cb9-f79c-49a3-9e8e-cfc988360df7" (UID: "fb410cb9-f79c-49a3-9e8e-cfc988360df7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:42:09.427594 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:09.427558    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzpvt\" (UniqueName: \"kubernetes.io/projected/8f2796a7-8378-4dc7-84f0-59ef3a6e9666-kube-api-access-vzpvt\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:42:09.427594 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:09.427589    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb410cb9-f79c-49a3-9e8e-cfc988360df7-util\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:42:09.427594 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:09.427602    2565 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f2796a7-8378-4dc7-84f0-59ef3a6e9666-util\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:42:09.427866 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:09.427610    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f2796a7-8378-4dc7-84f0-59ef3a6e9666-bundle\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:42:09.427866 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:09.427619    2565 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb410cb9-f79c-49a3-9e8e-cfc988360df7-bundle\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:42:09.427866 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:09.427627    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npd6v\" (UniqueName: \"kubernetes.io/projected/fb410cb9-f79c-49a3-9e8e-cfc988360df7-kube-api-access-npd6v\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:42:10.054336 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:10.054289    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6"
Jul 08 16:42:10.054336 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:10.054325    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/299160ebe81f164e400242457c5a1580453e7d84fd8087b72a6eb317efbn8s6" event={"ID":"8f2796a7-8378-4dc7-84f0-59ef3a6e9666","Type":"ContainerDied","Data":"6e78357be9d71557d333dd30f4094c662bc19b1193846c37f4dad585ecae7b5a"}
Jul 08 16:42:10.054823 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:10.054377    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e78357be9d71557d333dd30f4094c662bc19b1193846c37f4dad585ecae7b5a"
Jul 08 16:42:10.056220 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:10.056200    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4"
Jul 08 16:42:10.056352 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:10.056194    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/f13fb8b568e19b60b32189bed3ab51e2b7cfe62efa490862cd57eead76t27n4" event={"ID":"fb410cb9-f79c-49a3-9e8e-cfc988360df7","Type":"ContainerDied","Data":"89f477af01cef91206efa40994bdd8367ed09b11ffb2e467e95cef93cafadc35"}
Jul 08 16:42:10.056352 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:10.056321    2565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89f477af01cef91206efa40994bdd8367ed09b11ffb2e467e95cef93cafadc35"
Jul 08 16:42:13.080960 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:13.080914    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:15.021440 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.021401    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-dx5b4"]
Jul 08 16:42:15.021841 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.021773    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad1ecd44-907c-4d52-91b9-14442cce8a85" containerName="extract"
Jul 08 16:42:15.021841 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.021787    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad1ecd44-907c-4d52-91b9-14442cce8a85" containerName="extract"
Jul 08 16:42:15.021841 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.021802    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad1ecd44-907c-4d52-91b9-14442cce8a85" containerName="util"
Jul 08 16:42:15.021841 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.021812    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad1ecd44-907c-4d52-91b9-14442cce8a85" containerName="util"
Jul 08 16:42:15.021841 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.021822    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb410cb9-f79c-49a3-9e8e-cfc988360df7" containerName="util"
Jul 08 16:42:15.021841 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.021831    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb410cb9-f79c-49a3-9e8e-cfc988360df7" containerName="util"
Jul 08 16:42:15.022073 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.021850    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb410cb9-f79c-49a3-9e8e-cfc988360df7" containerName="extract"
Jul 08 16:42:15.022073 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.021859    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb410cb9-f79c-49a3-9e8e-cfc988360df7" containerName="extract"
Jul 08 16:42:15.022073 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.021871    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb410cb9-f79c-49a3-9e8e-cfc988360df7" containerName="pull"
Jul 08 16:42:15.022073 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.021879    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb410cb9-f79c-49a3-9e8e-cfc988360df7" containerName="pull"
Jul 08 16:42:15.022073 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.021890    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="827dd06e-8dc6-4598-a094-e9e7ae1d220d" containerName="pull"
Jul 08 16:42:15.022073 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.021898    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="827dd06e-8dc6-4598-a094-e9e7ae1d220d" containerName="pull"
Jul 08 16:42:15.022073 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.021909    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f2796a7-8378-4dc7-84f0-59ef3a6e9666" containerName="pull"
Jul 08 16:42:15.022073 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.021917    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f2796a7-8378-4dc7-84f0-59ef3a6e9666" containerName="pull"
Jul 08 16:42:15.022073 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.021929    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f2796a7-8378-4dc7-84f0-59ef3a6e9666" containerName="extract"
Jul 08 16:42:15.022073 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.021937    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f2796a7-8378-4dc7-84f0-59ef3a6e9666" containerName="extract"
Jul 08 16:42:15.022073 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.021948    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad1ecd44-907c-4d52-91b9-14442cce8a85" containerName="pull"
Jul 08 16:42:15.022073 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.021957    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad1ecd44-907c-4d52-91b9-14442cce8a85" containerName="pull"
Jul 08 16:42:15.022073 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.021979    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="827dd06e-8dc6-4598-a094-e9e7ae1d220d" containerName="extract"
Jul 08 16:42:15.022073 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.021990    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="827dd06e-8dc6-4598-a094-e9e7ae1d220d" containerName="extract"
Jul 08 16:42:15.022073 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.022003    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="827dd06e-8dc6-4598-a094-e9e7ae1d220d" containerName="util"
Jul 08 16:42:15.022073 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.022017    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="827dd06e-8dc6-4598-a094-e9e7ae1d220d" containerName="util"
Jul 08 16:42:15.022073 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.022031    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f2796a7-8378-4dc7-84f0-59ef3a6e9666" containerName="util"
Jul 08 16:42:15.022073 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.022042    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f2796a7-8378-4dc7-84f0-59ef3a6e9666" containerName="util"
Jul 08 16:42:15.022626 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.022157    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb410cb9-f79c-49a3-9e8e-cfc988360df7" containerName="extract"
Jul 08 16:42:15.022626 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.022170    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="827dd06e-8dc6-4598-a094-e9e7ae1d220d" containerName="extract"
Jul 08 16:42:15.022626 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.022183    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad1ecd44-907c-4d52-91b9-14442cce8a85" containerName="extract"
Jul 08 16:42:15.022626 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.022192    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f2796a7-8378-4dc7-84f0-59ef3a6e9666" containerName="extract"
Jul 08 16:42:15.026601 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.026581    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-dx5b4"
Jul 08 16:42:15.031779 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.031752    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 08 16:42:15.031779 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.031771    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-8fl6k\""
Jul 08 16:42:15.031990 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.031828    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 08 16:42:15.031990 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.031860    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 08 16:42:15.040428 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.040402    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-dx5b4"]
Jul 08 16:42:15.181613 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.181577    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsg4d\" (UniqueName: \"kubernetes.io/projected/525ed767-7832-403b-899f-1443f130dadb-kube-api-access-vsg4d\") pod \"dns-operator-controller-manager-65b49595d7-dx5b4\" (UID: \"525ed767-7832-403b-899f-1443f130dadb\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-dx5b4"
Jul 08 16:42:15.272007 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.271917    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:15.282984 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.282959    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsg4d\" (UniqueName: \"kubernetes.io/projected/525ed767-7832-403b-899f-1443f130dadb-kube-api-access-vsg4d\") pod \"dns-operator-controller-manager-65b49595d7-dx5b4\" (UID: \"525ed767-7832-403b-899f-1443f130dadb\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-dx5b4"
Jul 08 16:42:15.292447 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.292408    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsg4d\" (UniqueName: \"kubernetes.io/projected/525ed767-7832-403b-899f-1443f130dadb-kube-api-access-vsg4d\") pod \"dns-operator-controller-manager-65b49595d7-dx5b4\" (UID: \"525ed767-7832-403b-899f-1443f130dadb\") " pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-dx5b4"
Jul 08 16:42:15.338617 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.338580    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-dx5b4"
Jul 08 16:42:15.510331 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.510282    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-65b49595d7-dx5b4"]
Jul 08 16:42:15.510622 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:42:15.510601    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod525ed767_7832_403b_899f_1443f130dadb.slice/crio-ed89731f72f6dddd711cb5cf5eeca755c210c4605990b223fcdb98b680c8f843 WatchSource:0}: Error finding container ed89731f72f6dddd711cb5cf5eeca755c210c4605990b223fcdb98b680c8f843: Status 404 returned error can't find the container with id ed89731f72f6dddd711cb5cf5eeca755c210c4605990b223fcdb98b680c8f843
Jul 08 16:42:15.591364 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.591270    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:15.727300 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.727260    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:15.884325 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:15.884229    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:16.087602 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:16.087569    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-dx5b4" event={"ID":"525ed767-7832-403b-899f-1443f130dadb","Type":"ContainerStarted","Data":"ed89731f72f6dddd711cb5cf5eeca755c210c4605990b223fcdb98b680c8f843"}
Jul 08 16:42:16.815584 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:16.815544    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:16.943914 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:16.943858    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85d9d9b968-h2892"]
Jul 08 16:42:16.950806 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:16.950776    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85d9d9b968-h2892"
Jul 08 16:42:16.969745 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:16.969711    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85d9d9b968-h2892"]
Jul 08 16:42:17.099650 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:17.099558    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a2587bc-4630-45e3-b984-5c917d0301e7-oauth-serving-cert\") pod \"console-85d9d9b968-h2892\" (UID: \"9a2587bc-4630-45e3-b984-5c917d0301e7\") " pod="openshift-console/console-85d9d9b968-h2892"
Jul 08 16:42:17.099650 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:17.099602    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a2587bc-4630-45e3-b984-5c917d0301e7-console-serving-cert\") pod \"console-85d9d9b968-h2892\" (UID: \"9a2587bc-4630-45e3-b984-5c917d0301e7\") " pod="openshift-console/console-85d9d9b968-h2892"
Jul 08 16:42:17.100094 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:17.099699    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2587bc-4630-45e3-b984-5c917d0301e7-service-ca\") pod \"console-85d9d9b968-h2892\" (UID: \"9a2587bc-4630-45e3-b984-5c917d0301e7\") " pod="openshift-console/console-85d9d9b968-h2892"
Jul 08 16:42:17.100094 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:17.099754    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a2587bc-4630-45e3-b984-5c917d0301e7-console-config\") pod \"console-85d9d9b968-h2892\" (UID: \"9a2587bc-4630-45e3-b984-5c917d0301e7\") " pod="openshift-console/console-85d9d9b968-h2892"
Jul 08 16:42:17.100094 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:17.099794    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gw77\" (UniqueName: \"kubernetes.io/projected/9a2587bc-4630-45e3-b984-5c917d0301e7-kube-api-access-6gw77\") pod \"console-85d9d9b968-h2892\" (UID: \"9a2587bc-4630-45e3-b984-5c917d0301e7\") " pod="openshift-console/console-85d9d9b968-h2892"
Jul 08 16:42:17.100094 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:17.099819    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a2587bc-4630-45e3-b984-5c917d0301e7-console-oauth-config\") pod \"console-85d9d9b968-h2892\" (UID: \"9a2587bc-4630-45e3-b984-5c917d0301e7\") " pod="openshift-console/console-85d9d9b968-h2892"
Jul 08 16:42:17.100094 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:17.099923    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a2587bc-4630-45e3-b984-5c917d0301e7-trusted-ca-bundle\") pod \"console-85d9d9b968-h2892\" (UID: \"9a2587bc-4630-45e3-b984-5c917d0301e7\") " pod="openshift-console/console-85d9d9b968-h2892"
Jul 08 16:42:17.201158 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:17.201118    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a2587bc-4630-45e3-b984-5c917d0301e7-trusted-ca-bundle\") pod \"console-85d9d9b968-h2892\" (UID: \"9a2587bc-4630-45e3-b984-5c917d0301e7\") " pod="openshift-console/console-85d9d9b968-h2892"
Jul 08 16:42:17.201352 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:17.201177    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a2587bc-4630-45e3-b984-5c917d0301e7-oauth-serving-cert\") pod \"console-85d9d9b968-h2892\" (UID: \"9a2587bc-4630-45e3-b984-5c917d0301e7\") " pod="openshift-console/console-85d9d9b968-h2892"
Jul 08 16:42:17.201352 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:17.201207    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a2587bc-4630-45e3-b984-5c917d0301e7-console-serving-cert\") pod \"console-85d9d9b968-h2892\" (UID: \"9a2587bc-4630-45e3-b984-5c917d0301e7\") " pod="openshift-console/console-85d9d9b968-h2892"
Jul 08 16:42:17.201352 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:17.201254    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2587bc-4630-45e3-b984-5c917d0301e7-service-ca\") pod \"console-85d9d9b968-h2892\" (UID: \"9a2587bc-4630-45e3-b984-5c917d0301e7\") " pod="openshift-console/console-85d9d9b968-h2892"
Jul 08 16:42:17.201352 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:17.201312    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a2587bc-4630-45e3-b984-5c917d0301e7-console-config\") pod \"console-85d9d9b968-h2892\" (UID: \"9a2587bc-4630-45e3-b984-5c917d0301e7\") " pod="openshift-console/console-85d9d9b968-h2892"
Jul 08 16:42:17.201576 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:17.201363    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gw77\" (UniqueName: \"kubernetes.io/projected/9a2587bc-4630-45e3-b984-5c917d0301e7-kube-api-access-6gw77\") pod \"console-85d9d9b968-h2892\" (UID: \"9a2587bc-4630-45e3-b984-5c917d0301e7\") " pod="openshift-console/console-85d9d9b968-h2892"
Jul 08 16:42:17.201576 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:17.201391    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a2587bc-4630-45e3-b984-5c917d0301e7-console-oauth-config\") pod \"console-85d9d9b968-h2892\" (UID: \"9a2587bc-4630-45e3-b984-5c917d0301e7\") " pod="openshift-console/console-85d9d9b968-h2892"
Jul 08 16:42:17.202105 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:17.202076    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9a2587bc-4630-45e3-b984-5c917d0301e7-oauth-serving-cert\") pod \"console-85d9d9b968-h2892\" (UID: \"9a2587bc-4630-45e3-b984-5c917d0301e7\") " pod="openshift-console/console-85d9d9b968-h2892"
Jul 08 16:42:17.202221 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:17.202168    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9a2587bc-4630-45e3-b984-5c917d0301e7-trusted-ca-bundle\") pod \"console-85d9d9b968-h2892\" (UID: \"9a2587bc-4630-45e3-b984-5c917d0301e7\") " pod="openshift-console/console-85d9d9b968-h2892"
Jul 08 16:42:17.202277 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:17.202261    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9a2587bc-4630-45e3-b984-5c917d0301e7-service-ca\") pod \"console-85d9d9b968-h2892\" (UID: \"9a2587bc-4630-45e3-b984-5c917d0301e7\") " pod="openshift-console/console-85d9d9b968-h2892"
Jul 08 16:42:17.202370 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:17.202348    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9a2587bc-4630-45e3-b984-5c917d0301e7-console-config\") pod \"console-85d9d9b968-h2892\" (UID: \"9a2587bc-4630-45e3-b984-5c917d0301e7\") " pod="openshift-console/console-85d9d9b968-h2892"
Jul 08 16:42:17.204184 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:17.204162    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a2587bc-4630-45e3-b984-5c917d0301e7-console-serving-cert\") pod \"console-85d9d9b968-h2892\" (UID: \"9a2587bc-4630-45e3-b984-5c917d0301e7\") " pod="openshift-console/console-85d9d9b968-h2892"
Jul 08 16:42:17.204306 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:17.204257    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9a2587bc-4630-45e3-b984-5c917d0301e7-console-oauth-config\") pod \"console-85d9d9b968-h2892\" (UID: \"9a2587bc-4630-45e3-b984-5c917d0301e7\") " pod="openshift-console/console-85d9d9b968-h2892"
Jul 08 16:42:17.220229 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:17.220191    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gw77\" (UniqueName: \"kubernetes.io/projected/9a2587bc-4630-45e3-b984-5c917d0301e7-kube-api-access-6gw77\") pod \"console-85d9d9b968-h2892\" (UID: \"9a2587bc-4630-45e3-b984-5c917d0301e7\") " pod="openshift-console/console-85d9d9b968-h2892"
Jul 08 16:42:17.261796 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:17.261754    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85d9d9b968-h2892"
Jul 08 16:42:17.560743 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:17.556647    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85d9d9b968-h2892"]
Jul 08 16:42:18.101485 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:18.101437    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-dx5b4" event={"ID":"525ed767-7832-403b-899f-1443f130dadb","Type":"ContainerStarted","Data":"a01d03cdac6ad0ea5a07ac3c1073f9057ff69ca9a0fe1ab08824d24a2a99a712"}
Jul 08 16:42:18.101964 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:18.101548    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-dx5b4"
Jul 08 16:42:18.102938 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:18.102916    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85d9d9b968-h2892" event={"ID":"9a2587bc-4630-45e3-b984-5c917d0301e7","Type":"ContainerStarted","Data":"a63106ff6b1efe801b4383f7b06f8cc428c63087781146995566de71e50faf44"}
Jul 08 16:42:18.103012 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:18.102956    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85d9d9b968-h2892" event={"ID":"9a2587bc-4630-45e3-b984-5c917d0301e7","Type":"ContainerStarted","Data":"b507c235457bd7b1bc959d43d91bc034ee88fba4d3b2070adf4049cd7e5117cf"}
Jul 08 16:42:18.146993 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:18.146929    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-dx5b4" podStartSLOduration=2.181424888 podStartE2EDuration="4.146909826s" totalImagesPullingTime="1.965484938s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:42:18.146393838 +0000 UTC m=+474.859755287" watchObservedRunningTime="2026-07-08 16:42:18.146909826 +0000 UTC m=+474.860271253"
Jul 08 16:42:18.177330 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:18.177293    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-6d85f6564-hp27q"]
Jul 08 16:42:18.180741 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:18.180718    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6d85f6564-hp27q"
Jul 08 16:42:18.183802 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:18.183772    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-rxnmx\""
Jul 08 16:42:18.210166 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:18.210106    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85d9d9b968-h2892" podStartSLOduration=2.210087022 podStartE2EDuration="2.210087022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:42:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:42:18.206549492 +0000 UTC m=+474.919910920" watchObservedRunningTime="2026-07-08 16:42:18.210087022 +0000 UTC m=+474.923448452"
Jul 08 16:42:18.214039 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:18.214009    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6d85f6564-hp27q"]
Jul 08 16:42:18.312712 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:18.312654    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn59q\" (UniqueName: \"kubernetes.io/projected/a74002bb-6e21-4774-88a0-60f3e557ac54-kube-api-access-bn59q\") pod \"authorino-operator-6d85f6564-hp27q\" (UID: \"a74002bb-6e21-4774-88a0-60f3e557ac54\") " pod="kuadrant-system/authorino-operator-6d85f6564-hp27q"
Jul 08 16:42:18.414044 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:18.413947    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn59q\" (UniqueName: \"kubernetes.io/projected/a74002bb-6e21-4774-88a0-60f3e557ac54-kube-api-access-bn59q\") pod \"authorino-operator-6d85f6564-hp27q\" (UID: \"a74002bb-6e21-4774-88a0-60f3e557ac54\") " pod="kuadrant-system/authorino-operator-6d85f6564-hp27q"
Jul 08 16:42:18.424141 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:18.424107    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn59q\" (UniqueName: \"kubernetes.io/projected/a74002bb-6e21-4774-88a0-60f3e557ac54-kube-api-access-bn59q\") pod \"authorino-operator-6d85f6564-hp27q\" (UID: \"a74002bb-6e21-4774-88a0-60f3e557ac54\") " pod="kuadrant-system/authorino-operator-6d85f6564-hp27q"
Jul 08 16:42:18.491601 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:18.491564    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-6d85f6564-hp27q"
Jul 08 16:42:18.651784 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:18.651741    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-6d85f6564-hp27q"]
Jul 08 16:42:18.655021 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:42:18.654951    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda74002bb_6e21_4774_88a0_60f3e557ac54.slice/crio-d0ebfef7a46455d89a90923ccdc07864b5ecb195aa850baf8c32eb23f17cc066 WatchSource:0}: Error finding container d0ebfef7a46455d89a90923ccdc07864b5ecb195aa850baf8c32eb23f17cc066: Status 404 returned error can't find the container with id d0ebfef7a46455d89a90923ccdc07864b5ecb195aa850baf8c32eb23f17cc066
Jul 08 16:42:18.785643 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:18.785607    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:18.893292 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:18.893251    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:19.079152 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:19.079058    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:19.112263 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:19.112197    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6d85f6564-hp27q" event={"ID":"a74002bb-6e21-4774-88a0-60f3e557ac54","Type":"ContainerStarted","Data":"d0ebfef7a46455d89a90923ccdc07864b5ecb195aa850baf8c32eb23f17cc066"}
Jul 08 16:42:19.490171 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:19.490110    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:20.404202 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:20.404165    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:20.516258 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:20.516209    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:21.126786 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:21.126699    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-6d85f6564-hp27q" event={"ID":"a74002bb-6e21-4774-88a0-60f3e557ac54","Type":"ContainerStarted","Data":"868de6bd5ed43f41d968fb93834ec2fbf6cf40fd1a5b6e9c7c76f5507d8092e4"}
Jul 08 16:42:21.126940 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:21.126837    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-6d85f6564-hp27q"
Jul 08 16:42:21.183284 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:21.183210    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-6d85f6564-hp27q" podStartSLOduration=1.040942204 podStartE2EDuration="3.183190219s" totalImagesPullingTime="2.142248015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:42:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:42:21.182333675 +0000 UTC m=+477.895695117" watchObservedRunningTime="2026-07-08 16:42:21.183190219 +0000 UTC m=+477.896551645"
Jul 08 16:42:22.905315 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:22.905279    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:23.144273 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:23.144241    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tldlw"]
Jul 08 16:42:23.147801 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:23.147781    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tldlw"
Jul 08 16:42:23.150556 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:23.150536    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-cw4w6\""
Jul 08 16:42:23.165597 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:23.165535    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tldlw"]
Jul 08 16:42:23.260244 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:23.260207    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7a29723-2210-4ce6-acfe-711776cdb84c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-tldlw\" (UID: \"e7a29723-2210-4ce6-acfe-711776cdb84c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tldlw"
Jul 08 16:42:23.260244 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:23.260248    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcgn4\" (UniqueName: \"kubernetes.io/projected/e7a29723-2210-4ce6-acfe-711776cdb84c-kube-api-access-mcgn4\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-tldlw\" (UID: \"e7a29723-2210-4ce6-acfe-711776cdb84c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tldlw"
Jul 08 16:42:23.361889 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:23.361852    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7a29723-2210-4ce6-acfe-711776cdb84c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-tldlw\" (UID: \"e7a29723-2210-4ce6-acfe-711776cdb84c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tldlw"
Jul 08 16:42:23.362080 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:23.361902    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcgn4\" (UniqueName: \"kubernetes.io/projected/e7a29723-2210-4ce6-acfe-711776cdb84c-kube-api-access-mcgn4\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-tldlw\" (UID: \"e7a29723-2210-4ce6-acfe-711776cdb84c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tldlw"
Jul 08 16:42:23.362466 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:23.362435    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e7a29723-2210-4ce6-acfe-711776cdb84c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-tldlw\" (UID: \"e7a29723-2210-4ce6-acfe-711776cdb84c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tldlw"
Jul 08 16:42:23.376701 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:23.376649    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcgn4\" (UniqueName: \"kubernetes.io/projected/e7a29723-2210-4ce6-acfe-711776cdb84c-kube-api-access-mcgn4\") pod \"kuadrant-operator-controller-manager-bb7ddb7d7-tldlw\" (UID: \"e7a29723-2210-4ce6-acfe-711776cdb84c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tldlw"
Jul 08 16:42:23.458135 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:23.458091    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tldlw"
Jul 08 16:42:23.629017 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:23.628980    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tldlw"]
Jul 08 16:42:23.650621 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:42:23.650545    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7a29723_2210_4ce6_acfe_711776cdb84c.slice/crio-54abb9ef713417b0f737937b49f17bd3af9d57904bd1cf7028fb0286c0faf609 WatchSource:0}: Error finding container 54abb9ef713417b0f737937b49f17bd3af9d57904bd1cf7028fb0286c0faf609: Status 404 returned error can't find the container with id 54abb9ef713417b0f737937b49f17bd3af9d57904bd1cf7028fb0286c0faf609
Jul 08 16:42:24.142886 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:24.142843    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tldlw" event={"ID":"e7a29723-2210-4ce6-acfe-711776cdb84c","Type":"ContainerStarted","Data":"54abb9ef713417b0f737937b49f17bd3af9d57904bd1cf7028fb0286c0faf609"}
Jul 08 16:42:25.321890 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:25.321851    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:25.422342 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:25.422290    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:25.551355 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:25.551282    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:27.262468 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:27.262434    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85d9d9b968-h2892"
Jul 08 16:42:27.262930 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:27.262503    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85d9d9b968-h2892"
Jul 08 16:42:27.268254 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:27.268226    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85d9d9b968-h2892"
Jul 08 16:42:28.167888 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:28.167835    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tldlw" event={"ID":"e7a29723-2210-4ce6-acfe-711776cdb84c","Type":"ContainerStarted","Data":"7253834ca5bd695cca60d38dc38c84558f51461a3704623b8cc18ebee443c809"}
Jul 08 16:42:28.168091 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:28.167910    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tldlw"
Jul 08 16:42:28.172141 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:28.172115    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85d9d9b968-h2892"
Jul 08 16:42:28.195213 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:28.195164    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tldlw" podStartSLOduration=1.487615516 podStartE2EDuration="5.195147147s" totalImagesPullingTime="3.707531631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:42:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:42:28.192435111 +0000 UTC m=+484.905796602" watchObservedRunningTime="2026-07-08 16:42:28.195147147 +0000 UTC m=+484.908508589"
Jul 08 16:42:28.259180 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:28.259147    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cd65d986f-5vl7z"]
Jul 08 16:42:28.376242 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:28.376211    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:28.481300 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:28.481261    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:28.544263 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:28.544219    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:29.114909 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:29.114880    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-65b49595d7-dx5b4"
Jul 08 16:42:30.025143 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:30.025104    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:30.198105 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:30.198069    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:30.371172 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:30.371074    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:30.515608 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:30.515572    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:30.703885 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:30.703847    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:30.804979 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:30.804943    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:32.134211 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:32.134173    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-6d85f6564-hp27q"
Jul 08 16:42:35.754670 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:35.754636    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:35.849261 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:35.849225    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:35.949466 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:35.949430    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:39.176182 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:39.176154    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-bb7ddb7d7-tldlw"
Jul 08 16:42:42.807824 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:42.807784    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:42.910843 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:42.910808    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:45.324916 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:45.324876    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:47.194103 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:47.194070    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:49.979365 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:49.979320    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:50.080142 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:50.080096    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:53.284244 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:53.284192    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-6cd65d986f-5vl7z" podUID="fe1b9009-6574-409f-8e18-7a7eb8146dc5" containerName="console" containerID="cri-o://18e0d13489b65332ae264bd50af684faee2ebddbcb7d1c7b5b5243b5bb4494a5" gracePeriod=15
Jul 08 16:42:53.537502 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:53.537434    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd65d986f-5vl7z"
Jul 08 16:42:53.634932 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:53.634876    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe1b9009-6574-409f-8e18-7a7eb8146dc5-oauth-serving-cert\") pod \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\" (UID: \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\") "
Jul 08 16:42:53.635228 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:53.634971    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe1b9009-6574-409f-8e18-7a7eb8146dc5-trusted-ca-bundle\") pod \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\" (UID: \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\") "
Jul 08 16:42:53.635228 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:53.635035    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j86b4\" (UniqueName: \"kubernetes.io/projected/fe1b9009-6574-409f-8e18-7a7eb8146dc5-kube-api-access-j86b4\") pod \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\" (UID: \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\") "
Jul 08 16:42:53.635228 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:53.635094    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe1b9009-6574-409f-8e18-7a7eb8146dc5-service-ca\") pod \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\" (UID: \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\") "
Jul 08 16:42:53.635228 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:53.635122    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe1b9009-6574-409f-8e18-7a7eb8146dc5-console-oauth-config\") pod \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\" (UID: \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\") "
Jul 08 16:42:53.635444 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:53.635238    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe1b9009-6574-409f-8e18-7a7eb8146dc5-console-serving-cert\") pod \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\" (UID: \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\") "
Jul 08 16:42:53.635444 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:53.635339    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe1b9009-6574-409f-8e18-7a7eb8146dc5-console-config\") pod \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\" (UID: \"fe1b9009-6574-409f-8e18-7a7eb8146dc5\") "
Jul 08 16:42:53.635550 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:53.635436    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe1b9009-6574-409f-8e18-7a7eb8146dc5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fe1b9009-6574-409f-8e18-7a7eb8146dc5" (UID: "fe1b9009-6574-409f-8e18-7a7eb8146dc5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:42:53.635550 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:53.635481    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe1b9009-6574-409f-8e18-7a7eb8146dc5-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fe1b9009-6574-409f-8e18-7a7eb8146dc5" (UID: "fe1b9009-6574-409f-8e18-7a7eb8146dc5"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:42:53.635782 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:53.635605    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe1b9009-6574-409f-8e18-7a7eb8146dc5-service-ca" (OuterVolumeSpecName: "service-ca") pod "fe1b9009-6574-409f-8e18-7a7eb8146dc5" (UID: "fe1b9009-6574-409f-8e18-7a7eb8146dc5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:42:53.635942 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:53.635860    2565 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe1b9009-6574-409f-8e18-7a7eb8146dc5-trusted-ca-bundle\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:42:53.635942 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:53.635876    2565 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe1b9009-6574-409f-8e18-7a7eb8146dc5-service-ca\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:42:53.635942 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:53.635891    2565 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe1b9009-6574-409f-8e18-7a7eb8146dc5-oauth-serving-cert\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:42:53.635942 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:53.635927    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe1b9009-6574-409f-8e18-7a7eb8146dc5-console-config" (OuterVolumeSpecName: "console-config") pod "fe1b9009-6574-409f-8e18-7a7eb8146dc5" (UID: "fe1b9009-6574-409f-8e18-7a7eb8146dc5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:42:53.637965 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:53.637932    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe1b9009-6574-409f-8e18-7a7eb8146dc5-kube-api-access-j86b4" (OuterVolumeSpecName: "kube-api-access-j86b4") pod "fe1b9009-6574-409f-8e18-7a7eb8146dc5" (UID: "fe1b9009-6574-409f-8e18-7a7eb8146dc5"). InnerVolumeSpecName "kube-api-access-j86b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:42:53.638236 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:53.638197    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe1b9009-6574-409f-8e18-7a7eb8146dc5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fe1b9009-6574-409f-8e18-7a7eb8146dc5" (UID: "fe1b9009-6574-409f-8e18-7a7eb8146dc5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:42:53.638389 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:53.638226    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe1b9009-6574-409f-8e18-7a7eb8146dc5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fe1b9009-6574-409f-8e18-7a7eb8146dc5" (UID: "fe1b9009-6574-409f-8e18-7a7eb8146dc5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:42:53.737169 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:53.737120    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j86b4\" (UniqueName: \"kubernetes.io/projected/fe1b9009-6574-409f-8e18-7a7eb8146dc5-kube-api-access-j86b4\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:42:53.737169 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:53.737159    2565 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe1b9009-6574-409f-8e18-7a7eb8146dc5-console-oauth-config\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:42:53.737169 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:53.737169    2565 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe1b9009-6574-409f-8e18-7a7eb8146dc5-console-serving-cert\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:42:53.737169 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:53.737182    2565 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe1b9009-6574-409f-8e18-7a7eb8146dc5-console-config\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:42:54.311003 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:54.310950    2565 generic.go:356] "Generic (PLEG): container finished" podID="fe1b9009-6574-409f-8e18-7a7eb8146dc5" containerID="18e0d13489b65332ae264bd50af684faee2ebddbcb7d1c7b5b5243b5bb4494a5" exitCode=0
Jul 08 16:42:54.311589 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:54.311045    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd65d986f-5vl7z"
Jul 08 16:42:54.311788 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:54.311096    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd65d986f-5vl7z" event={"ID":"fe1b9009-6574-409f-8e18-7a7eb8146dc5","Type":"ContainerDied","Data":"18e0d13489b65332ae264bd50af684faee2ebddbcb7d1c7b5b5243b5bb4494a5"}
Jul 08 16:42:54.311901 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:54.311875    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd65d986f-5vl7z" event={"ID":"fe1b9009-6574-409f-8e18-7a7eb8146dc5","Type":"ContainerDied","Data":"ffcde9be8254bf69cc218b4cd39ce5fcabe5f3453bc96b4ccb50d4963e7be6e7"}
Jul 08 16:42:54.311958 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:54.311937    2565 scope.go:117] "RemoveContainer" containerID="18e0d13489b65332ae264bd50af684faee2ebddbcb7d1c7b5b5243b5bb4494a5"
Jul 08 16:42:54.327104 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:54.327081    2565 scope.go:117] "RemoveContainer" containerID="18e0d13489b65332ae264bd50af684faee2ebddbcb7d1c7b5b5243b5bb4494a5"
Jul 08 16:42:54.327410 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:42:54.327390    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18e0d13489b65332ae264bd50af684faee2ebddbcb7d1c7b5b5243b5bb4494a5\": container with ID starting with 18e0d13489b65332ae264bd50af684faee2ebddbcb7d1c7b5b5243b5bb4494a5 not found: ID does not exist" containerID="18e0d13489b65332ae264bd50af684faee2ebddbcb7d1c7b5b5243b5bb4494a5"
Jul 08 16:42:54.327461 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:54.327418    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18e0d13489b65332ae264bd50af684faee2ebddbcb7d1c7b5b5243b5bb4494a5"} err="failed to get container status \"18e0d13489b65332ae264bd50af684faee2ebddbcb7d1c7b5b5243b5bb4494a5\": rpc error: code = NotFound desc = could not find container \"18e0d13489b65332ae264bd50af684faee2ebddbcb7d1c7b5b5243b5bb4494a5\": container with ID starting with 18e0d13489b65332ae264bd50af684faee2ebddbcb7d1c7b5b5243b5bb4494a5 not found: ID does not exist"
Jul 08 16:42:54.336912 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:54.336884    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cd65d986f-5vl7z"]
Jul 08 16:42:54.342927 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:54.342900    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cd65d986f-5vl7z"]
Jul 08 16:42:54.746996 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:54.746957    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:54.875947 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:54.875905    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:54.976315 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:54.976278    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:42:55.822393 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:42:55.822358    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe1b9009-6574-409f-8e18-7a7eb8146dc5" path="/var/lib/kubelet/pods/fe1b9009-6574-409f-8e18-7a7eb8146dc5/volumes"
Jul 08 16:43:01.761936 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:01.761896    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:01.859250 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:01.859217    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:04.505326 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:04.505291    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:04.599103 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:04.599071    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:09.776177 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:09.776137    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:09.845094 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:09.845060    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:10.162704 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:10.162601    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:10.257942 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:10.257909    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:10.354174 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:10.354135    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:10.354670 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:10.354652    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-lx9ww"]
Jul 08 16:43:10.355043 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:10.355030    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe1b9009-6574-409f-8e18-7a7eb8146dc5" containerName="console"
Jul 08 16:43:10.355084 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:10.355045    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe1b9009-6574-409f-8e18-7a7eb8146dc5" containerName="console"
Jul 08 16:43:10.355117 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:10.355107    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe1b9009-6574-409f-8e18-7a7eb8146dc5" containerName="console"
Jul 08 16:43:10.357890 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:10.357871    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-lx9ww"
Jul 08 16:43:10.360290 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:10.360268    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-sthpr\""
Jul 08 16:43:10.361157 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:10.361130    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 08 16:43:10.368535 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:10.368506    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-lx9ww"]
Jul 08 16:43:10.444392 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:10.444362    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-lx9ww"]
Jul 08 16:43:10.446786 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:10.446737    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:10.477835 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:10.477810    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg8n2\" (UniqueName: \"kubernetes.io/projected/44793ca2-0576-431d-a64e-0181df1cfcaf-kube-api-access-mg8n2\") pod \"limitador-limitador-ff68ff5cb-lx9ww\" (UID: \"44793ca2-0576-431d-a64e-0181df1cfcaf\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-lx9ww"
Jul 08 16:43:10.477983 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:10.477861    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/44793ca2-0576-431d-a64e-0181df1cfcaf-config-file\") pod \"limitador-limitador-ff68ff5cb-lx9ww\" (UID: \"44793ca2-0576-431d-a64e-0181df1cfcaf\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-lx9ww"
Jul 08 16:43:10.516075 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:10.516042    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:10.579314 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:10.579280    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg8n2\" (UniqueName: \"kubernetes.io/projected/44793ca2-0576-431d-a64e-0181df1cfcaf-kube-api-access-mg8n2\") pod \"limitador-limitador-ff68ff5cb-lx9ww\" (UID: \"44793ca2-0576-431d-a64e-0181df1cfcaf\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-lx9ww"
Jul 08 16:43:10.579660 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:10.579329    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/44793ca2-0576-431d-a64e-0181df1cfcaf-config-file\") pod \"limitador-limitador-ff68ff5cb-lx9ww\" (UID: \"44793ca2-0576-431d-a64e-0181df1cfcaf\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-lx9ww"
Jul 08 16:43:10.580016 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:10.579998    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/44793ca2-0576-431d-a64e-0181df1cfcaf-config-file\") pod \"limitador-limitador-ff68ff5cb-lx9ww\" (UID: \"44793ca2-0576-431d-a64e-0181df1cfcaf\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-lx9ww"
Jul 08 16:43:10.589053 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:10.589032    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg8n2\" (UniqueName: \"kubernetes.io/projected/44793ca2-0576-431d-a64e-0181df1cfcaf-kube-api-access-mg8n2\") pod \"limitador-limitador-ff68ff5cb-lx9ww\" (UID: \"44793ca2-0576-431d-a64e-0181df1cfcaf\") " pod="kuadrant-system/limitador-limitador-ff68ff5cb-lx9ww"
Jul 08 16:43:10.668652 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:10.668620    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-lx9ww"
Jul 08 16:43:10.808473 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:10.808438    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-lx9ww"]
Jul 08 16:43:10.809315 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:43:10.809292    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44793ca2_0576_431d_a64e_0181df1cfcaf.slice/crio-3f171370c03d0c2cb867933ebb87dae49e345296cf180889cd44a87f24cdaddd WatchSource:0}: Error finding container 3f171370c03d0c2cb867933ebb87dae49e345296cf180889cd44a87f24cdaddd: Status 404 returned error can't find the container with id 3f171370c03d0c2cb867933ebb87dae49e345296cf180889cd44a87f24cdaddd
Jul 08 16:43:11.276255 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:11.276214    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:11.372854 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:11.372820    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:11.400909 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:11.400877    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-lx9ww" event={"ID":"44793ca2-0576-431d-a64e-0181df1cfcaf","Type":"ContainerStarted","Data":"3f171370c03d0c2cb867933ebb87dae49e345296cf180889cd44a87f24cdaddd"}
Jul 08 16:43:11.423034 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:11.423003    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-8df697c9f-zjlcz"]
Jul 08 16:43:11.426008 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:11.425992    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-zjlcz"
Jul 08 16:43:11.428917 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:11.428898    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-9kjm9\""
Jul 08 16:43:11.435238 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:11.435219    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-zjlcz"]
Jul 08 16:43:11.588788 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:11.588701    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vsbr\" (UniqueName: \"kubernetes.io/projected/094deb0b-eda9-4dec-8c3f-a2a5d9f71e22-kube-api-access-4vsbr\") pod \"authorino-8df697c9f-zjlcz\" (UID: \"094deb0b-eda9-4dec-8c3f-a2a5d9f71e22\") " pod="kuadrant-system/authorino-8df697c9f-zjlcz"
Jul 08 16:43:11.689768 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:11.689730    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vsbr\" (UniqueName: \"kubernetes.io/projected/094deb0b-eda9-4dec-8c3f-a2a5d9f71e22-kube-api-access-4vsbr\") pod \"authorino-8df697c9f-zjlcz\" (UID: \"094deb0b-eda9-4dec-8c3f-a2a5d9f71e22\") " pod="kuadrant-system/authorino-8df697c9f-zjlcz"
Jul 08 16:43:11.700024 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:11.699998    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vsbr\" (UniqueName: \"kubernetes.io/projected/094deb0b-eda9-4dec-8c3f-a2a5d9f71e22-kube-api-access-4vsbr\") pod \"authorino-8df697c9f-zjlcz\" (UID: \"094deb0b-eda9-4dec-8c3f-a2a5d9f71e22\") " pod="kuadrant-system/authorino-8df697c9f-zjlcz"
Jul 08 16:43:11.736792 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:11.736762    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-zjlcz"
Jul 08 16:43:11.878450 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:11.878378    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-8df697c9f-zjlcz"]
Jul 08 16:43:11.881363 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:43:11.881331    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod094deb0b_eda9_4dec_8c3f_a2a5d9f71e22.slice/crio-d40800330a912c562167618619abbe5708b2bae99eff67cda456aaf15196db47 WatchSource:0}: Error finding container d40800330a912c562167618619abbe5708b2bae99eff67cda456aaf15196db47: Status 404 returned error can't find the container with id d40800330a912c562167618619abbe5708b2bae99eff67cda456aaf15196db47
Jul 08 16:43:12.408743 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:12.408704    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-zjlcz" event={"ID":"094deb0b-eda9-4dec-8c3f-a2a5d9f71e22","Type":"ContainerStarted","Data":"d40800330a912c562167618619abbe5708b2bae99eff67cda456aaf15196db47"}
Jul 08 16:43:14.611977 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:14.611941    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:14.710988 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:14.710957    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:14.811802 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:14.811751    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:14.912139 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:14.912055    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:17.442926 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:17.442870    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-zjlcz" event={"ID":"094deb0b-eda9-4dec-8c3f-a2a5d9f71e22","Type":"ContainerStarted","Data":"5c3a3e63464234d849e637535feced61f31b12d7272e535e7fda77a2f46d6c29"}
Jul 08 16:43:17.444574 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:17.444551    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-lx9ww" event={"ID":"44793ca2-0576-431d-a64e-0181df1cfcaf","Type":"ContainerStarted","Data":"ed6cdc8c9802eccbbf1a6cf3bb9d29baab9bc8a81ed63a0ceac2c7d7984534ec"}
Jul 08 16:43:17.444719 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:17.444701    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-lx9ww"
Jul 08 16:43:17.461293 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:17.461237    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-8df697c9f-zjlcz" podStartSLOduration=1.268464899 podStartE2EDuration="6.461221058s" totalImagesPullingTime="5.192756159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:43:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:43:17.459706477 +0000 UTC m=+534.173067901" watchObservedRunningTime="2026-07-08 16:43:17.461221058 +0000 UTC m=+534.174582476"
Jul 08 16:43:17.481118 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:17.481067    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-ff68ff5cb-lx9ww" podStartSLOduration=1.155502563 podStartE2EDuration="7.48104856s" totalImagesPullingTime="6.325545997s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:43:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:43:17.47838856 +0000 UTC m=+534.191750009" watchObservedRunningTime="2026-07-08 16:43:17.48104856 +0000 UTC m=+534.194409988"
Jul 08 16:43:21.653610 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:21.653575    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:23.101068 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:23.101034    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:23.163355 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:23.163325    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:26.558211 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:26.558175    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-lx9ww"]
Jul 08 16:43:26.558651 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:26.558360    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-ff68ff5cb-lx9ww" podUID="44793ca2-0576-431d-a64e-0181df1cfcaf" containerName="limitador" containerID="cri-o://ed6cdc8c9802eccbbf1a6cf3bb9d29baab9bc8a81ed63a0ceac2c7d7984534ec" gracePeriod=30
Jul 08 16:43:26.560439 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:26.560411    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-ff68ff5cb-lx9ww"
Jul 08 16:43:27.114304 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:27.114264    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-lx9ww"
Jul 08 16:43:27.126024 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:27.125945    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/44793ca2-0576-431d-a64e-0181df1cfcaf-config-file\") pod \"44793ca2-0576-431d-a64e-0181df1cfcaf\" (UID: \"44793ca2-0576-431d-a64e-0181df1cfcaf\") "
Jul 08 16:43:27.126024 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:27.126019    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mg8n2\" (UniqueName: \"kubernetes.io/projected/44793ca2-0576-431d-a64e-0181df1cfcaf-kube-api-access-mg8n2\") pod \"44793ca2-0576-431d-a64e-0181df1cfcaf\" (UID: \"44793ca2-0576-431d-a64e-0181df1cfcaf\") "
Jul 08 16:43:27.126407 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:27.126373    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44793ca2-0576-431d-a64e-0181df1cfcaf-config-file" (OuterVolumeSpecName: "config-file") pod "44793ca2-0576-431d-a64e-0181df1cfcaf" (UID: "44793ca2-0576-431d-a64e-0181df1cfcaf"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:43:27.128366 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:27.128322    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44793ca2-0576-431d-a64e-0181df1cfcaf-kube-api-access-mg8n2" (OuterVolumeSpecName: "kube-api-access-mg8n2") pod "44793ca2-0576-431d-a64e-0181df1cfcaf" (UID: "44793ca2-0576-431d-a64e-0181df1cfcaf"). InnerVolumeSpecName "kube-api-access-mg8n2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:43:27.226976 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:27.226925    2565 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/44793ca2-0576-431d-a64e-0181df1cfcaf-config-file\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:43:27.226976 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:27.226966    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg8n2\" (UniqueName: \"kubernetes.io/projected/44793ca2-0576-431d-a64e-0181df1cfcaf-kube-api-access-mg8n2\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:43:27.496909 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:27.496873    2565 generic.go:356] "Generic (PLEG): container finished" podID="44793ca2-0576-431d-a64e-0181df1cfcaf" containerID="ed6cdc8c9802eccbbf1a6cf3bb9d29baab9bc8a81ed63a0ceac2c7d7984534ec" exitCode=0
Jul 08 16:43:27.497107 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:27.496937    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-ff68ff5cb-lx9ww"
Jul 08 16:43:27.497107 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:27.496967    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-lx9ww" event={"ID":"44793ca2-0576-431d-a64e-0181df1cfcaf","Type":"ContainerDied","Data":"ed6cdc8c9802eccbbf1a6cf3bb9d29baab9bc8a81ed63a0ceac2c7d7984534ec"}
Jul 08 16:43:27.497107 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:27.497058    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-ff68ff5cb-lx9ww" event={"ID":"44793ca2-0576-431d-a64e-0181df1cfcaf","Type":"ContainerDied","Data":"3f171370c03d0c2cb867933ebb87dae49e345296cf180889cd44a87f24cdaddd"}
Jul 08 16:43:27.497107 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:27.497082    2565 scope.go:117] "RemoveContainer" containerID="ed6cdc8c9802eccbbf1a6cf3bb9d29baab9bc8a81ed63a0ceac2c7d7984534ec"
Jul 08 16:43:27.506786 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:27.506760    2565 scope.go:117] "RemoveContainer" containerID="ed6cdc8c9802eccbbf1a6cf3bb9d29baab9bc8a81ed63a0ceac2c7d7984534ec"
Jul 08 16:43:27.507142 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:43:27.507116    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed6cdc8c9802eccbbf1a6cf3bb9d29baab9bc8a81ed63a0ceac2c7d7984534ec\": container with ID starting with ed6cdc8c9802eccbbf1a6cf3bb9d29baab9bc8a81ed63a0ceac2c7d7984534ec not found: ID does not exist" containerID="ed6cdc8c9802eccbbf1a6cf3bb9d29baab9bc8a81ed63a0ceac2c7d7984534ec"
Jul 08 16:43:27.507217 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:27.507183    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed6cdc8c9802eccbbf1a6cf3bb9d29baab9bc8a81ed63a0ceac2c7d7984534ec"} err="failed to get container status \"ed6cdc8c9802eccbbf1a6cf3bb9d29baab9bc8a81ed63a0ceac2c7d7984534ec\": rpc error: code = NotFound desc = could not find container \"ed6cdc8c9802eccbbf1a6cf3bb9d29baab9bc8a81ed63a0ceac2c7d7984534ec\": container with ID starting with ed6cdc8c9802eccbbf1a6cf3bb9d29baab9bc8a81ed63a0ceac2c7d7984534ec not found: ID does not exist"
Jul 08 16:43:27.523786 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:27.523745    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-lx9ww"]
Jul 08 16:43:27.525862 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:27.525831    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-ff68ff5cb-lx9ww"]
Jul 08 16:43:27.823577 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:27.823463    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44793ca2-0576-431d-a64e-0181df1cfcaf" path="/var/lib/kubelet/pods/44793ca2-0576-431d-a64e-0181df1cfcaf/volumes"
Jul 08 16:43:32.586841 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:32.586801    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:33.453941 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:33.453900    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:43:36.377524 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:36.377482    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7c77957575-lkzck"]
Jul 08 16:43:36.377940 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:36.377865    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44793ca2-0576-431d-a64e-0181df1cfcaf" containerName="limitador"
Jul 08 16:43:36.377940 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:36.377877    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="44793ca2-0576-431d-a64e-0181df1cfcaf" containerName="limitador"
Jul 08 16:43:36.377940 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:36.377931    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="44793ca2-0576-431d-a64e-0181df1cfcaf" containerName="limitador"
Jul 08 16:43:36.382270 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:36.382247    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-lkzck"
Jul 08 16:43:36.385478 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:36.385453    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 08 16:43:36.391451 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:36.391418    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-lkzck"]
Jul 08 16:43:36.407394 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:36.407354    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b74fcdd0-709a-4fc2-bd48-8a6b345b1f92-tls-cert\") pod \"authorino-7c77957575-lkzck\" (UID: \"b74fcdd0-709a-4fc2-bd48-8a6b345b1f92\") " pod="kuadrant-system/authorino-7c77957575-lkzck"
Jul 08 16:43:36.407581 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:36.407479    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl7r5\" (UniqueName: \"kubernetes.io/projected/b74fcdd0-709a-4fc2-bd48-8a6b345b1f92-kube-api-access-xl7r5\") pod \"authorino-7c77957575-lkzck\" (UID: \"b74fcdd0-709a-4fc2-bd48-8a6b345b1f92\") " pod="kuadrant-system/authorino-7c77957575-lkzck"
Jul 08 16:43:36.508768 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:36.508720    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl7r5\" (UniqueName: \"kubernetes.io/projected/b74fcdd0-709a-4fc2-bd48-8a6b345b1f92-kube-api-access-xl7r5\") pod \"authorino-7c77957575-lkzck\" (UID: \"b74fcdd0-709a-4fc2-bd48-8a6b345b1f92\") " pod="kuadrant-system/authorino-7c77957575-lkzck"
Jul 08 16:43:36.508989 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:36.508797    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b74fcdd0-709a-4fc2-bd48-8a6b345b1f92-tls-cert\") pod \"authorino-7c77957575-lkzck\" (UID: \"b74fcdd0-709a-4fc2-bd48-8a6b345b1f92\") " pod="kuadrant-system/authorino-7c77957575-lkzck"
Jul 08 16:43:36.511450 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:36.511417    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b74fcdd0-709a-4fc2-bd48-8a6b345b1f92-tls-cert\") pod \"authorino-7c77957575-lkzck\" (UID: \"b74fcdd0-709a-4fc2-bd48-8a6b345b1f92\") " pod="kuadrant-system/authorino-7c77957575-lkzck"
Jul 08 16:43:36.525193 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:36.525151    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl7r5\" (UniqueName: \"kubernetes.io/projected/b74fcdd0-709a-4fc2-bd48-8a6b345b1f92-kube-api-access-xl7r5\") pod \"authorino-7c77957575-lkzck\" (UID: \"b74fcdd0-709a-4fc2-bd48-8a6b345b1f92\") " pod="kuadrant-system/authorino-7c77957575-lkzck"
Jul 08 16:43:36.693047 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:36.693004    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7c77957575-lkzck"
Jul 08 16:43:36.841237 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:36.841206    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7c77957575-lkzck"]
Jul 08 16:43:36.843027 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:43:36.842997    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb74fcdd0_709a_4fc2_bd48_8a6b345b1f92.slice/crio-413e514cdcd6ce392bc7a2d040ef928c883db431ea61a4f8d27b9d9b7c8b8c87 WatchSource:0}: Error finding container 413e514cdcd6ce392bc7a2d040ef928c883db431ea61a4f8d27b9d9b7c8b8c87: Status 404 returned error can't find the container with id 413e514cdcd6ce392bc7a2d040ef928c883db431ea61a4f8d27b9d9b7c8b8c87
Jul 08 16:43:37.555613 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:37.555518    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-lkzck" event={"ID":"b74fcdd0-709a-4fc2-bd48-8a6b345b1f92","Type":"ContainerStarted","Data":"e0c389f2bd685f190a6552f3bed8e59cacdbe1077d264e9391d9a1e31a72a05c"}
Jul 08 16:43:37.555613 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:37.555566    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7c77957575-lkzck" event={"ID":"b74fcdd0-709a-4fc2-bd48-8a6b345b1f92","Type":"ContainerStarted","Data":"413e514cdcd6ce392bc7a2d040ef928c883db431ea61a4f8d27b9d9b7c8b8c87"}
Jul 08 16:43:37.577259 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:37.577203    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7c77957575-lkzck" podStartSLOduration=1.247956692 podStartE2EDuration="1.577184899s" totalImagesPullingTime="329.228207ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:43:37.574846657 +0000 UTC m=+554.288208104" watchObservedRunningTime="2026-07-08 16:43:37.577184899 +0000 UTC m=+554.290546367"
Jul 08 16:43:37.602845 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:37.602795    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-zjlcz"]
Jul 08 16:43:37.603034 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:37.603009    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-8df697c9f-zjlcz" podUID="094deb0b-eda9-4dec-8c3f-a2a5d9f71e22" containerName="authorino" containerID="cri-o://5c3a3e63464234d849e637535feced61f31b12d7272e535e7fda77a2f46d6c29" gracePeriod=30
Jul 08 16:43:37.861463 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:37.861436    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-zjlcz"
Jul 08 16:43:37.920874 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:37.920841    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vsbr\" (UniqueName: \"kubernetes.io/projected/094deb0b-eda9-4dec-8c3f-a2a5d9f71e22-kube-api-access-4vsbr\") pod \"094deb0b-eda9-4dec-8c3f-a2a5d9f71e22\" (UID: \"094deb0b-eda9-4dec-8c3f-a2a5d9f71e22\") "
Jul 08 16:43:37.922970 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:37.922937    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094deb0b-eda9-4dec-8c3f-a2a5d9f71e22-kube-api-access-4vsbr" (OuterVolumeSpecName: "kube-api-access-4vsbr") pod "094deb0b-eda9-4dec-8c3f-a2a5d9f71e22" (UID: "094deb0b-eda9-4dec-8c3f-a2a5d9f71e22"). InnerVolumeSpecName "kube-api-access-4vsbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:43:38.021793 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:38.021747    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vsbr\" (UniqueName: \"kubernetes.io/projected/094deb0b-eda9-4dec-8c3f-a2a5d9f71e22-kube-api-access-4vsbr\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:43:38.562281 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:38.562241    2565 generic.go:356] "Generic (PLEG): container finished" podID="094deb0b-eda9-4dec-8c3f-a2a5d9f71e22" containerID="5c3a3e63464234d849e637535feced61f31b12d7272e535e7fda77a2f46d6c29" exitCode=0
Jul 08 16:43:38.562281 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:38.562287    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-8df697c9f-zjlcz"
Jul 08 16:43:38.562877 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:38.562343    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-zjlcz" event={"ID":"094deb0b-eda9-4dec-8c3f-a2a5d9f71e22","Type":"ContainerDied","Data":"5c3a3e63464234d849e637535feced61f31b12d7272e535e7fda77a2f46d6c29"}
Jul 08 16:43:38.562877 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:38.562388    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-8df697c9f-zjlcz" event={"ID":"094deb0b-eda9-4dec-8c3f-a2a5d9f71e22","Type":"ContainerDied","Data":"d40800330a912c562167618619abbe5708b2bae99eff67cda456aaf15196db47"}
Jul 08 16:43:38.562877 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:38.562410    2565 scope.go:117] "RemoveContainer" containerID="5c3a3e63464234d849e637535feced61f31b12d7272e535e7fda77a2f46d6c29"
Jul 08 16:43:38.572243 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:38.572221    2565 scope.go:117] "RemoveContainer" containerID="5c3a3e63464234d849e637535feced61f31b12d7272e535e7fda77a2f46d6c29"
Jul 08 16:43:38.572539 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:43:38.572517    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c3a3e63464234d849e637535feced61f31b12d7272e535e7fda77a2f46d6c29\": container with ID starting with 5c3a3e63464234d849e637535feced61f31b12d7272e535e7fda77a2f46d6c29 not found: ID does not exist" containerID="5c3a3e63464234d849e637535feced61f31b12d7272e535e7fda77a2f46d6c29"
Jul 08 16:43:38.572608 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:38.572547    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c3a3e63464234d849e637535feced61f31b12d7272e535e7fda77a2f46d6c29"} err="failed to get container status \"5c3a3e63464234d849e637535feced61f31b12d7272e535e7fda77a2f46d6c29\": rpc error: code = NotFound desc = could not find container \"5c3a3e63464234d849e637535feced61f31b12d7272e535e7fda77a2f46d6c29\": container with ID starting with 5c3a3e63464234d849e637535feced61f31b12d7272e535e7fda77a2f46d6c29 not found: ID does not exist"
Jul 08 16:43:38.586712 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:38.586657    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-8df697c9f-zjlcz"]
Jul 08 16:43:38.591174 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:38.591136    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-8df697c9f-zjlcz"]
Jul 08 16:43:39.822416 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:39.822373    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="094deb0b-eda9-4dec-8c3f-a2a5d9f71e22" path="/var/lib/kubelet/pods/094deb0b-eda9-4dec-8c3f-a2a5d9f71e22/volumes"
Jul 08 16:43:42.085778 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.085739    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-687b7449bc-5hzqb"]
Jul 08 16:43:42.086168 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.085940    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve/kserve-controller-manager-687b7449bc-5hzqb" podUID="b4e253de-313d-4de6-8e27-cc24d13b355c" containerName="manager" containerID="cri-o://448934ec7eda4c2e7810a83f180d69c1f50cf84448f1337ffca8b1a4d487e4f7" gracePeriod=10
Jul 08 16:43:42.117732 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.117693    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-687b7449bc-w4q6g"]
Jul 08 16:43:42.118083 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.118070    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="094deb0b-eda9-4dec-8c3f-a2a5d9f71e22" containerName="authorino"
Jul 08 16:43:42.118132 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.118085    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="094deb0b-eda9-4dec-8c3f-a2a5d9f71e22" containerName="authorino"
Jul 08 16:43:42.118168 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.118139    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="094deb0b-eda9-4dec-8c3f-a2a5d9f71e22" containerName="authorino"
Jul 08 16:43:42.121167 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.121144    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-687b7449bc-w4q6g"
Jul 08 16:43:42.130396 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.130365    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-687b7449bc-w4q6g"]
Jul 08 16:43:42.158725 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.158688    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abab7113-b73a-446b-bff7-f48652ba2e7e-cert\") pod \"kserve-controller-manager-687b7449bc-w4q6g\" (UID: \"abab7113-b73a-446b-bff7-f48652ba2e7e\") " pod="kserve/kserve-controller-manager-687b7449bc-w4q6g"
Jul 08 16:43:42.158910 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.158767    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdd9b\" (UniqueName: \"kubernetes.io/projected/abab7113-b73a-446b-bff7-f48652ba2e7e-kube-api-access-wdd9b\") pod \"kserve-controller-manager-687b7449bc-w4q6g\" (UID: \"abab7113-b73a-446b-bff7-f48652ba2e7e\") " pod="kserve/kserve-controller-manager-687b7449bc-w4q6g"
Jul 08 16:43:42.259579 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.259157    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdd9b\" (UniqueName: \"kubernetes.io/projected/abab7113-b73a-446b-bff7-f48652ba2e7e-kube-api-access-wdd9b\") pod \"kserve-controller-manager-687b7449bc-w4q6g\" (UID: \"abab7113-b73a-446b-bff7-f48652ba2e7e\") " pod="kserve/kserve-controller-manager-687b7449bc-w4q6g"
Jul 08 16:43:42.259579 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.259251    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abab7113-b73a-446b-bff7-f48652ba2e7e-cert\") pod \"kserve-controller-manager-687b7449bc-w4q6g\" (UID: \"abab7113-b73a-446b-bff7-f48652ba2e7e\") " pod="kserve/kserve-controller-manager-687b7449bc-w4q6g"
Jul 08 16:43:42.261792 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.261758    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/abab7113-b73a-446b-bff7-f48652ba2e7e-cert\") pod \"kserve-controller-manager-687b7449bc-w4q6g\" (UID: \"abab7113-b73a-446b-bff7-f48652ba2e7e\") " pod="kserve/kserve-controller-manager-687b7449bc-w4q6g"
Jul 08 16:43:42.270200 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.270172    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdd9b\" (UniqueName: \"kubernetes.io/projected/abab7113-b73a-446b-bff7-f48652ba2e7e-kube-api-access-wdd9b\") pod \"kserve-controller-manager-687b7449bc-w4q6g\" (UID: \"abab7113-b73a-446b-bff7-f48652ba2e7e\") " pod="kserve/kserve-controller-manager-687b7449bc-w4q6g"
Jul 08 16:43:42.334446 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.334419    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-687b7449bc-5hzqb"
Jul 08 16:43:42.461424 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.461380    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9t58\" (UniqueName: \"kubernetes.io/projected/b4e253de-313d-4de6-8e27-cc24d13b355c-kube-api-access-j9t58\") pod \"b4e253de-313d-4de6-8e27-cc24d13b355c\" (UID: \"b4e253de-313d-4de6-8e27-cc24d13b355c\") "
Jul 08 16:43:42.461582 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.461443    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4e253de-313d-4de6-8e27-cc24d13b355c-cert\") pod \"b4e253de-313d-4de6-8e27-cc24d13b355c\" (UID: \"b4e253de-313d-4de6-8e27-cc24d13b355c\") "
Jul 08 16:43:42.463696 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.463643    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4e253de-313d-4de6-8e27-cc24d13b355c-cert" (OuterVolumeSpecName: "cert") pod "b4e253de-313d-4de6-8e27-cc24d13b355c" (UID: "b4e253de-313d-4de6-8e27-cc24d13b355c"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:43:42.463811 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.463727    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4e253de-313d-4de6-8e27-cc24d13b355c-kube-api-access-j9t58" (OuterVolumeSpecName: "kube-api-access-j9t58") pod "b4e253de-313d-4de6-8e27-cc24d13b355c" (UID: "b4e253de-313d-4de6-8e27-cc24d13b355c"). InnerVolumeSpecName "kube-api-access-j9t58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:43:42.469879 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.469858    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-687b7449bc-w4q6g"
Jul 08 16:43:42.562388 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.562350    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9t58\" (UniqueName: \"kubernetes.io/projected/b4e253de-313d-4de6-8e27-cc24d13b355c-kube-api-access-j9t58\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:43:42.562388 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.562391    2565 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b4e253de-313d-4de6-8e27-cc24d13b355c-cert\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:43:42.588540 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.588504    2565 generic.go:356] "Generic (PLEG): container finished" podID="b4e253de-313d-4de6-8e27-cc24d13b355c" containerID="448934ec7eda4c2e7810a83f180d69c1f50cf84448f1337ffca8b1a4d487e4f7" exitCode=0
Jul 08 16:43:42.588730 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.588553    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-687b7449bc-5hzqb"
Jul 08 16:43:42.588730 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.588580    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-687b7449bc-5hzqb" event={"ID":"b4e253de-313d-4de6-8e27-cc24d13b355c","Type":"ContainerDied","Data":"448934ec7eda4c2e7810a83f180d69c1f50cf84448f1337ffca8b1a4d487e4f7"}
Jul 08 16:43:42.588730 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.588635    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-687b7449bc-5hzqb" event={"ID":"b4e253de-313d-4de6-8e27-cc24d13b355c","Type":"ContainerDied","Data":"b4bc49e5cd51f8c3538063562afd188d747e4cbaaf31735373eb995f7ba8a0ab"}
Jul 08 16:43:42.588730 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.588665    2565 scope.go:117] "RemoveContainer" containerID="448934ec7eda4c2e7810a83f180d69c1f50cf84448f1337ffca8b1a4d487e4f7"
Jul 08 16:43:42.600089 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.600063    2565 scope.go:117] "RemoveContainer" containerID="448934ec7eda4c2e7810a83f180d69c1f50cf84448f1337ffca8b1a4d487e4f7"
Jul 08 16:43:42.600415 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:43:42.600395    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"448934ec7eda4c2e7810a83f180d69c1f50cf84448f1337ffca8b1a4d487e4f7\": container with ID starting with 448934ec7eda4c2e7810a83f180d69c1f50cf84448f1337ffca8b1a4d487e4f7 not found: ID does not exist" containerID="448934ec7eda4c2e7810a83f180d69c1f50cf84448f1337ffca8b1a4d487e4f7"
Jul 08 16:43:42.600503 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.600424    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"448934ec7eda4c2e7810a83f180d69c1f50cf84448f1337ffca8b1a4d487e4f7"} err="failed to get container status \"448934ec7eda4c2e7810a83f180d69c1f50cf84448f1337ffca8b1a4d487e4f7\": rpc error: code = NotFound desc = could not find container \"448934ec7eda4c2e7810a83f180d69c1f50cf84448f1337ffca8b1a4d487e4f7\": container with ID starting with 448934ec7eda4c2e7810a83f180d69c1f50cf84448f1337ffca8b1a4d487e4f7 not found: ID does not exist"
Jul 08 16:43:42.613436 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.613402    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-687b7449bc-5hzqb"]
Jul 08 16:43:42.617577 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.617551    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-687b7449bc-w4q6g"]
Jul 08 16:43:42.619868 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:43:42.619836    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabab7113_b73a_446b_bff7_f48652ba2e7e.slice/crio-cc47ef9a60d3be254a8367b085b098606074306def34fefade4a198d35301b6e WatchSource:0}: Error finding container cc47ef9a60d3be254a8367b085b098606074306def34fefade4a198d35301b6e: Status 404 returned error can't find the container with id cc47ef9a60d3be254a8367b085b098606074306def34fefade4a198d35301b6e
Jul 08 16:43:42.620598 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:42.620577    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-687b7449bc-5hzqb"]
Jul 08 16:43:43.595907 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:43.595867    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-687b7449bc-w4q6g" event={"ID":"abab7113-b73a-446b-bff7-f48652ba2e7e","Type":"ContainerStarted","Data":"cc47ef9a60d3be254a8367b085b098606074306def34fefade4a198d35301b6e"}
Jul 08 16:43:43.822384 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:43.822348    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4e253de-313d-4de6-8e27-cc24d13b355c" path="/var/lib/kubelet/pods/b4e253de-313d-4de6-8e27-cc24d13b355c/volumes"
Jul 08 16:43:44.604272 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:44.604228    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-687b7449bc-w4q6g" event={"ID":"abab7113-b73a-446b-bff7-f48652ba2e7e","Type":"ContainerStarted","Data":"8fbc10b663191d9a853b66ca1a5bf5809a5f99435f6af4be33d2d39a644d8828"}
Jul 08 16:43:44.604715 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:44.604333    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-687b7449bc-w4q6g"
Jul 08 16:43:44.625868 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:43:44.625749    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-687b7449bc-w4q6g" podStartSLOduration=0.891271671 podStartE2EDuration="2.625731311s" totalImagesPullingTime="1.73445964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:43:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:43:44.623254008 +0000 UTC m=+561.336615437" watchObservedRunningTime="2026-07-08 16:43:44.625731311 +0000 UTC m=+561.339092737"
Jul 08 16:44:11.062807 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:11.062669    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:44:11.165926 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:11.165844    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:44:15.619591 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:15.619519    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-687b7449bc-w4q6g"
Jul 08 16:44:16.723320 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:16.723270    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:44:16.724654 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:16.724610    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-qrsqp"]
Jul 08 16:44:16.725587 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:16.725542    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4e253de-313d-4de6-8e27-cc24d13b355c" containerName="manager"
Jul 08 16:44:16.725870 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:16.725798    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4e253de-313d-4de6-8e27-cc24d13b355c" containerName="manager"
Jul 08 16:44:16.726005 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:16.725988    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4e253de-313d-4de6-8e27-cc24d13b355c" containerName="manager"
Jul 08 16:44:16.738037 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:16.737989    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-qrsqp"
Jul 08 16:44:16.741060 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:16.741025    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 08 16:44:16.742383 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:16.742329    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-6x8rc\""
Jul 08 16:44:16.748618 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:16.748551    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-ps8wt"]
Jul 08 16:44:16.753605 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:16.752928    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-ps8wt"
Jul 08 16:44:16.753605 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:16.753377    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-qrsqp"]
Jul 08 16:44:16.756313 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:16.756263    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-lmkmd\""
Jul 08 16:44:16.756709 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:16.756665    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 08 16:44:16.765148 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:16.765096    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-ps8wt"]
Jul 08 16:44:16.802578 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:16.802517    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:44:16.859704 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:16.859373    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a234a1fb-078c-4982-b8c3-cab134ea4577-tls-certs\") pod \"model-serving-api-fd65d7d6-qrsqp\" (UID: \"a234a1fb-078c-4982-b8c3-cab134ea4577\") " pod="kserve/model-serving-api-fd65d7d6-qrsqp"
Jul 08 16:44:16.859704 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:16.859500    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp5v2\" (UniqueName: \"kubernetes.io/projected/a234a1fb-078c-4982-b8c3-cab134ea4577-kube-api-access-cp5v2\") pod \"model-serving-api-fd65d7d6-qrsqp\" (UID: \"a234a1fb-078c-4982-b8c3-cab134ea4577\") " pod="kserve/model-serving-api-fd65d7d6-qrsqp"
Jul 08 16:44:16.859704 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:16.859582    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f71f5666-76c2-46a0-a129-ceacedb14a92-cert\") pod \"odh-model-controller-84b54b7f97-ps8wt\" (UID: \"f71f5666-76c2-46a0-a129-ceacedb14a92\") " pod="kserve/odh-model-controller-84b54b7f97-ps8wt"
Jul 08 16:44:16.859704 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:16.859616    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2plq4\" (UniqueName: \"kubernetes.io/projected/f71f5666-76c2-46a0-a129-ceacedb14a92-kube-api-access-2plq4\") pod \"odh-model-controller-84b54b7f97-ps8wt\" (UID: \"f71f5666-76c2-46a0-a129-ceacedb14a92\") " pod="kserve/odh-model-controller-84b54b7f97-ps8wt"
Jul 08 16:44:16.960968 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:16.960883    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a234a1fb-078c-4982-b8c3-cab134ea4577-tls-certs\") pod \"model-serving-api-fd65d7d6-qrsqp\" (UID: \"a234a1fb-078c-4982-b8c3-cab134ea4577\") " pod="kserve/model-serving-api-fd65d7d6-qrsqp"
Jul 08 16:44:16.961382 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:16.961032    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp5v2\" (UniqueName: \"kubernetes.io/projected/a234a1fb-078c-4982-b8c3-cab134ea4577-kube-api-access-cp5v2\") pod \"model-serving-api-fd65d7d6-qrsqp\" (UID: \"a234a1fb-078c-4982-b8c3-cab134ea4577\") " pod="kserve/model-serving-api-fd65d7d6-qrsqp"
Jul 08 16:44:16.961382 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:16.961129    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f71f5666-76c2-46a0-a129-ceacedb14a92-cert\") pod \"odh-model-controller-84b54b7f97-ps8wt\" (UID: \"f71f5666-76c2-46a0-a129-ceacedb14a92\") " pod="kserve/odh-model-controller-84b54b7f97-ps8wt"
Jul 08 16:44:16.961382 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:16.961156    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2plq4\" (UniqueName: \"kubernetes.io/projected/f71f5666-76c2-46a0-a129-ceacedb14a92-kube-api-access-2plq4\") pod \"odh-model-controller-84b54b7f97-ps8wt\" (UID: \"f71f5666-76c2-46a0-a129-ceacedb14a92\") " pod="kserve/odh-model-controller-84b54b7f97-ps8wt"
Jul 08 16:44:16.961760 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:44:16.961724    2565 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 08 16:44:16.961879 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:44:16.961865    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a234a1fb-078c-4982-b8c3-cab134ea4577-tls-certs podName:a234a1fb-078c-4982-b8c3-cab134ea4577 nodeName:}" failed. No retries permitted until 2026-07-08 16:44:17.461826959 +0000 UTC m=+594.175188378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/a234a1fb-078c-4982-b8c3-cab134ea4577-tls-certs") pod "model-serving-api-fd65d7d6-qrsqp" (UID: "a234a1fb-078c-4982-b8c3-cab134ea4577") : secret "model-serving-api-tls" not found
Jul 08 16:44:16.961954 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:44:16.961878    2565 secret.go:189] Couldn't get secret kserve/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 08 16:44:16.961954 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:44:16.961932    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f71f5666-76c2-46a0-a129-ceacedb14a92-cert podName:f71f5666-76c2-46a0-a129-ceacedb14a92 nodeName:}" failed. No retries permitted until 2026-07-08 16:44:17.461920676 +0000 UTC m=+594.175282093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f71f5666-76c2-46a0-a129-ceacedb14a92-cert") pod "odh-model-controller-84b54b7f97-ps8wt" (UID: "f71f5666-76c2-46a0-a129-ceacedb14a92") : secret "odh-model-controller-webhook-cert" not found
Jul 08 16:44:16.975255 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:16.975114    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2plq4\" (UniqueName: \"kubernetes.io/projected/f71f5666-76c2-46a0-a129-ceacedb14a92-kube-api-access-2plq4\") pod \"odh-model-controller-84b54b7f97-ps8wt\" (UID: \"f71f5666-76c2-46a0-a129-ceacedb14a92\") " pod="kserve/odh-model-controller-84b54b7f97-ps8wt"
Jul 08 16:44:16.975255 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:16.975113    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp5v2\" (UniqueName: \"kubernetes.io/projected/a234a1fb-078c-4982-b8c3-cab134ea4577-kube-api-access-cp5v2\") pod \"model-serving-api-fd65d7d6-qrsqp\" (UID: \"a234a1fb-078c-4982-b8c3-cab134ea4577\") " pod="kserve/model-serving-api-fd65d7d6-qrsqp"
Jul 08 16:44:17.081134 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:17.081084    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:44:17.182901 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:17.182856    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:44:17.466547 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:17.466489    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f71f5666-76c2-46a0-a129-ceacedb14a92-cert\") pod \"odh-model-controller-84b54b7f97-ps8wt\" (UID: \"f71f5666-76c2-46a0-a129-ceacedb14a92\") " pod="kserve/odh-model-controller-84b54b7f97-ps8wt"
Jul 08 16:44:17.466894 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:17.466592    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a234a1fb-078c-4982-b8c3-cab134ea4577-tls-certs\") pod \"model-serving-api-fd65d7d6-qrsqp\" (UID: \"a234a1fb-078c-4982-b8c3-cab134ea4577\") " pod="kserve/model-serving-api-fd65d7d6-qrsqp"
Jul 08 16:44:17.469972 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:17.469927    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a234a1fb-078c-4982-b8c3-cab134ea4577-tls-certs\") pod \"model-serving-api-fd65d7d6-qrsqp\" (UID: \"a234a1fb-078c-4982-b8c3-cab134ea4577\") " pod="kserve/model-serving-api-fd65d7d6-qrsqp"
Jul 08 16:44:17.470177 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:17.469927    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f71f5666-76c2-46a0-a129-ceacedb14a92-cert\") pod \"odh-model-controller-84b54b7f97-ps8wt\" (UID: \"f71f5666-76c2-46a0-a129-ceacedb14a92\") " pod="kserve/odh-model-controller-84b54b7f97-ps8wt"
Jul 08 16:44:17.658004 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:17.657950    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-qrsqp"
Jul 08 16:44:17.681705 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:17.681635    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-ps8wt"
Jul 08 16:44:17.842384 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:17.841520    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-qrsqp"]
Jul 08 16:44:17.869792 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:17.869737    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-qrsqp" event={"ID":"a234a1fb-078c-4982-b8c3-cab134ea4577","Type":"ContainerStarted","Data":"e4f1b35ee8ff22393a60a7df7b25ee34d8dd950a8b1c26f86b089f0aa31b15ea"}
Jul 08 16:44:17.893400 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:17.893326    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-ps8wt"]
Jul 08 16:44:17.898313 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:44:17.898236    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71f5666_76c2_46a0_a129_ceacedb14a92.slice/crio-77832ece114adf57d76f69ea93b635636675facf083f47066f4dcee1c9acd72c WatchSource:0}: Error finding container 77832ece114adf57d76f69ea93b635636675facf083f47066f4dcee1c9acd72c: Status 404 returned error can't find the container with id 77832ece114adf57d76f69ea93b635636675facf083f47066f4dcee1c9acd72c
Jul 08 16:44:18.561804 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:18.561738    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:44:18.639751 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:18.639570    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:44:18.882616 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:18.882512    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-ps8wt" event={"ID":"f71f5666-76c2-46a0-a129-ceacedb14a92","Type":"ContainerStarted","Data":"77832ece114adf57d76f69ea93b635636675facf083f47066f4dcee1c9acd72c"}
Jul 08 16:44:20.898138 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:20.898102    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-qrsqp" event={"ID":"a234a1fb-078c-4982-b8c3-cab134ea4577","Type":"ContainerStarted","Data":"8584ddb75f8e903630cddefe48f41deb29dd8989b1b40ff6a50a4b745eab0ada"}
Jul 08 16:44:20.898589 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:20.898212    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-qrsqp"
Jul 08 16:44:20.899871 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:20.899846    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-ps8wt" event={"ID":"f71f5666-76c2-46a0-a129-ceacedb14a92","Type":"ContainerStarted","Data":"1be9978296dc084bd015f9160517f0db36323c49f69260248efff9c419f5f394"}
Jul 08 16:44:20.900020 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:20.899967    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-ps8wt"
Jul 08 16:44:20.926199 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:20.926144    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-qrsqp" podStartSLOduration=2.033430617 podStartE2EDuration="4.926125495s" totalImagesPullingTime="2.892694878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:44:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:44:20.92285451 +0000 UTC m=+597.636215968" watchObservedRunningTime="2026-07-08 16:44:20.926125495 +0000 UTC m=+597.639486968"
Jul 08 16:44:20.946519 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:20.946440    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-ps8wt" podStartSLOduration=2.105173733 podStartE2EDuration="4.946418299s" totalImagesPullingTime="2.841244566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:44:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:44:20.945754147 +0000 UTC m=+597.659115597" watchObservedRunningTime="2026-07-08 16:44:20.946418299 +0000 UTC m=+597.659779723"
Jul 08 16:44:30.610009 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:30.609966    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:44:31.908670 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:31.908633    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-ps8wt"
Jul 08 16:44:31.910584 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:31.910554    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-qrsqp"
Jul 08 16:44:37.902122 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:37.902078    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:44:58.755987 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:44:58.755804    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:45:42.244554 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:45:42.244511    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:45:52.925344 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:45:52.925296    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:45:53.026389 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:45:53.026344    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:05.277592 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.277541    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"]
Jul 08 16:46:05.281872 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.281831    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:05.286065 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.286022    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-dc4rb\""
Jul 08 16:46:05.286475 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.286451    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 08 16:46:05.287111 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.287087    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 16:46:05.287822 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.287791    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 16:46:05.308862 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.308808    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"]
Jul 08 16:46:05.327583 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.327530    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:05.339135 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.339052    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-6gdsh\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:05.339584 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.339150    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-6gdsh\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:05.339584 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.339223    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-6gdsh\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:05.339584 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.339310    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/207eee00-12dc-4b29-9e01-84874b8d403f-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-6gdsh\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:05.339584 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.339370    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-6gdsh\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:05.339584 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.339432    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-6gdsh\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:05.339584 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.339481    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/207eee00-12dc-4b29-9e01-84874b8d403f-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-6gdsh\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:05.339584 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.339529    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/207eee00-12dc-4b29-9e01-84874b8d403f-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-6gdsh\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:05.340055 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.339633    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghjhl\" (UniqueName: \"kubernetes.io/projected/207eee00-12dc-4b29-9e01-84874b8d403f-kube-api-access-ghjhl\") pod \"router-gateway-1-openshift-default-799f46c59b-6gdsh\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:05.419335 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.419260    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:05.440943 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.440873    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-6gdsh\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:05.440943 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.440936    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-6gdsh\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:05.441451 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.440977    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-6gdsh\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:05.441451 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.441037    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/207eee00-12dc-4b29-9e01-84874b8d403f-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-6gdsh\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:05.441451 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.441075    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-6gdsh\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:05.441451 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.441122    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-6gdsh\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:05.441451 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.441168    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/207eee00-12dc-4b29-9e01-84874b8d403f-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-6gdsh\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:05.441451 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.441214    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/207eee00-12dc-4b29-9e01-84874b8d403f-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-6gdsh\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:05.441451 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.441300    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghjhl\" (UniqueName: \"kubernetes.io/projected/207eee00-12dc-4b29-9e01-84874b8d403f-kube-api-access-ghjhl\") pod \"router-gateway-1-openshift-default-799f46c59b-6gdsh\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:05.441875 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.441739    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-6gdsh\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:05.442182 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.442133    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-6gdsh\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:05.442515 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.442248    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-6gdsh\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:05.442515 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.442306    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-6gdsh\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:05.442831 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.442780    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/207eee00-12dc-4b29-9e01-84874b8d403f-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-6gdsh\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:05.446564 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.446473    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-6gdsh\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:05.446930 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.446858    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/207eee00-12dc-4b29-9e01-84874b8d403f-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-6gdsh\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:05.460872 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.460789    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/207eee00-12dc-4b29-9e01-84874b8d403f-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-6gdsh\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:05.462946 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.462873    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghjhl\" (UniqueName: \"kubernetes.io/projected/207eee00-12dc-4b29-9e01-84874b8d403f-kube-api-access-ghjhl\") pod \"router-gateway-1-openshift-default-799f46c59b-6gdsh\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:05.517935 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.517830    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:05.598371 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.598306    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:05.812092 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.812044    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"]
Jul 08 16:46:05.815333 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:46:05.815292    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207eee00_12dc_4b29_9e01_84874b8d403f.slice/crio-87987b7d84b09c9a74ac814181e0bbfbecab208b6eca4d3cefe53f6b37b87bf9 WatchSource:0}: Error finding container 87987b7d84b09c9a74ac814181e0bbfbecab208b6eca4d3cefe53f6b37b87bf9: Status 404 returned error can't find the container with id 87987b7d84b09c9a74ac814181e0bbfbecab208b6eca4d3cefe53f6b37b87bf9
Jul 08 16:46:05.818140 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:05.818106    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 16:46:06.587052 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:06.587003    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh" event={"ID":"207eee00-12dc-4b29-9e01-84874b8d403f","Type":"ContainerStarted","Data":"87987b7d84b09c9a74ac814181e0bbfbecab208b6eca4d3cefe53f6b37b87bf9"}
Jul 08 16:46:08.681389 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:08.681303    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"]
Jul 08 16:46:08.764641 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:08.764558    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 08 16:46:08.764983 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:08.764781    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 08 16:46:08.764983 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:08.764864    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 08 16:46:09.617621 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:09.617581    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh" event={"ID":"207eee00-12dc-4b29-9e01-84874b8d403f","Type":"ContainerStarted","Data":"f25f3f65e949d543afa3ed04bba607dfba4ca0441659244449b81131ea4116e4"}
Jul 08 16:46:09.617859 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:09.617725    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh" podUID="207eee00-12dc-4b29-9e01-84874b8d403f" containerName="istio-proxy" containerID="cri-o://f25f3f65e949d543afa3ed04bba607dfba4ca0441659244449b81131ea4116e4" gracePeriod=30
Jul 08 16:46:09.629847 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:09.629801    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:09.647332 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:09.647261    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh" podStartSLOduration=1.7014755099999999 podStartE2EDuration="4.647238971s" totalImagesPullingTime="2.945763461s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:46:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:46:09.642378106 +0000 UTC m=+706.355739555" watchObservedRunningTime="2026-07-08 16:46:09.647238971 +0000 UTC m=+706.360600398"
Jul 08 16:46:14.903269 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:14.903221    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:15.054769 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.054608    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/207eee00-12dc-4b29-9e01-84874b8d403f-istio-token\") pod \"207eee00-12dc-4b29-9e01-84874b8d403f\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") "
Jul 08 16:46:15.054769 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.054706    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-istio-data\") pod \"207eee00-12dc-4b29-9e01-84874b8d403f\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") "
Jul 08 16:46:15.054769 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.054732    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/207eee00-12dc-4b29-9e01-84874b8d403f-istiod-ca-cert\") pod \"207eee00-12dc-4b29-9e01-84874b8d403f\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") "
Jul 08 16:46:15.055132 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.054784    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghjhl\" (UniqueName: \"kubernetes.io/projected/207eee00-12dc-4b29-9e01-84874b8d403f-kube-api-access-ghjhl\") pod \"207eee00-12dc-4b29-9e01-84874b8d403f\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") "
Jul 08 16:46:15.055132 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.054838    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/207eee00-12dc-4b29-9e01-84874b8d403f-istio-podinfo\") pod \"207eee00-12dc-4b29-9e01-84874b8d403f\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") "
Jul 08 16:46:15.055132 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.054883    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-workload-certs\") pod \"207eee00-12dc-4b29-9e01-84874b8d403f\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") "
Jul 08 16:46:15.055132 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.054976    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-credential-socket\") pod \"207eee00-12dc-4b29-9e01-84874b8d403f\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") "
Jul 08 16:46:15.055132 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.055008    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-istio-envoy\") pod \"207eee00-12dc-4b29-9e01-84874b8d403f\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") "
Jul 08 16:46:15.055132 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.055072    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-workload-socket\") pod \"207eee00-12dc-4b29-9e01-84874b8d403f\" (UID: \"207eee00-12dc-4b29-9e01-84874b8d403f\") "
Jul 08 16:46:15.055132 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.055081    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-istio-data" (OuterVolumeSpecName: "istio-data") pod "207eee00-12dc-4b29-9e01-84874b8d403f" (UID: "207eee00-12dc-4b29-9e01-84874b8d403f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:46:15.055491 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.055302    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "207eee00-12dc-4b29-9e01-84874b8d403f" (UID: "207eee00-12dc-4b29-9e01-84874b8d403f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:46:15.055491 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.055456    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-istio-data\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:46:15.055491 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.055475    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-workload-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:46:15.055491 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.055476    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "207eee00-12dc-4b29-9e01-84874b8d403f" (UID: "207eee00-12dc-4b29-9e01-84874b8d403f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:46:15.055715 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.055506    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "207eee00-12dc-4b29-9e01-84874b8d403f" (UID: "207eee00-12dc-4b29-9e01-84874b8d403f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:46:15.055715 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.055546    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/207eee00-12dc-4b29-9e01-84874b8d403f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "207eee00-12dc-4b29-9e01-84874b8d403f" (UID: "207eee00-12dc-4b29-9e01-84874b8d403f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:46:15.057947 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.057881    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/207eee00-12dc-4b29-9e01-84874b8d403f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "207eee00-12dc-4b29-9e01-84874b8d403f" (UID: "207eee00-12dc-4b29-9e01-84874b8d403f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 08 16:46:15.058155 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.057999    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "207eee00-12dc-4b29-9e01-84874b8d403f" (UID: "207eee00-12dc-4b29-9e01-84874b8d403f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:46:15.058155 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.058007    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/207eee00-12dc-4b29-9e01-84874b8d403f-kube-api-access-ghjhl" (OuterVolumeSpecName: "kube-api-access-ghjhl") pod "207eee00-12dc-4b29-9e01-84874b8d403f" (UID: "207eee00-12dc-4b29-9e01-84874b8d403f"). InnerVolumeSpecName "kube-api-access-ghjhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:46:15.058155 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.058010    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/207eee00-12dc-4b29-9e01-84874b8d403f-istio-token" (OuterVolumeSpecName: "istio-token") pod "207eee00-12dc-4b29-9e01-84874b8d403f" (UID: "207eee00-12dc-4b29-9e01-84874b8d403f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:46:15.156892 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.156757    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghjhl\" (UniqueName: \"kubernetes.io/projected/207eee00-12dc-4b29-9e01-84874b8d403f-kube-api-access-ghjhl\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:46:15.156892 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.156807    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/207eee00-12dc-4b29-9e01-84874b8d403f-istio-podinfo\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:46:15.156892 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.156824    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-credential-socket\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:46:15.156892 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.156841    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-istio-envoy\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:46:15.156892 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.156854    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/207eee00-12dc-4b29-9e01-84874b8d403f-workload-socket\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:46:15.156892 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.156868    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/207eee00-12dc-4b29-9e01-84874b8d403f-istio-token\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:46:15.156892 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.156883    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/207eee00-12dc-4b29-9e01-84874b8d403f-istiod-ca-cert\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:46:15.663377 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.663333    2565 generic.go:356] "Generic (PLEG): container finished" podID="207eee00-12dc-4b29-9e01-84874b8d403f" containerID="f25f3f65e949d543afa3ed04bba607dfba4ca0441659244449b81131ea4116e4" exitCode=0
Jul 08 16:46:15.663647 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.663389    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh" event={"ID":"207eee00-12dc-4b29-9e01-84874b8d403f","Type":"ContainerDied","Data":"f25f3f65e949d543afa3ed04bba607dfba4ca0441659244449b81131ea4116e4"}
Jul 08 16:46:15.663647 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.663531    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh" event={"ID":"207eee00-12dc-4b29-9e01-84874b8d403f","Type":"ContainerDied","Data":"87987b7d84b09c9a74ac814181e0bbfbecab208b6eca4d3cefe53f6b37b87bf9"}
Jul 08 16:46:15.663647 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.663569    2565 scope.go:117] "RemoveContainer" containerID="f25f3f65e949d543afa3ed04bba607dfba4ca0441659244449b81131ea4116e4"
Jul 08 16:46:15.664006 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.663974    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"
Jul 08 16:46:15.678852 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.678814    2565 scope.go:117] "RemoveContainer" containerID="f25f3f65e949d543afa3ed04bba607dfba4ca0441659244449b81131ea4116e4"
Jul 08 16:46:15.679354 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:46:15.679316    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f25f3f65e949d543afa3ed04bba607dfba4ca0441659244449b81131ea4116e4\": container with ID starting with f25f3f65e949d543afa3ed04bba607dfba4ca0441659244449b81131ea4116e4 not found: ID does not exist" containerID="f25f3f65e949d543afa3ed04bba607dfba4ca0441659244449b81131ea4116e4"
Jul 08 16:46:15.679591 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.679372    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f25f3f65e949d543afa3ed04bba607dfba4ca0441659244449b81131ea4116e4"} err="failed to get container status \"f25f3f65e949d543afa3ed04bba607dfba4ca0441659244449b81131ea4116e4\": rpc error: code = NotFound desc = could not find container \"f25f3f65e949d543afa3ed04bba607dfba4ca0441659244449b81131ea4116e4\": container with ID starting with f25f3f65e949d543afa3ed04bba607dfba4ca0441659244449b81131ea4116e4 not found: ID does not exist"
Jul 08 16:46:15.696001 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.695929    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"]
Jul 08 16:46:15.710999 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.710946    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-6gdsh"]
Jul 08 16:46:15.729775 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.729729    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:15.824126 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:15.824074    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="207eee00-12dc-4b29-9e01-84874b8d403f" path="/var/lib/kubelet/pods/207eee00-12dc-4b29-9e01-84874b8d403f/volumes"
Jul 08 16:46:18.362074 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.362020    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"]
Jul 08 16:46:18.362651 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.362575    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="207eee00-12dc-4b29-9e01-84874b8d403f" containerName="istio-proxy"
Jul 08 16:46:18.362651 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.362592    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="207eee00-12dc-4b29-9e01-84874b8d403f" containerName="istio-proxy"
Jul 08 16:46:18.362774 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.362692    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="207eee00-12dc-4b29-9e01-84874b8d403f" containerName="istio-proxy"
Jul 08 16:46:18.366341 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.366287    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:18.369163 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.369117    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 08 16:46:18.369712 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.369661    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-gw-ca-root-cert\""
Jul 08 16:46:18.370389 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.370360    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 08 16:46:18.370513 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.370368    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-1-openshift-default-dockercfg-dc4rb\""
Jul 08 16:46:18.384858 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.384811    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"]
Jul 08 16:46:18.500590 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.500536    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-dw428\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:18.500590 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.500597    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-dw428\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:18.501044 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.500718    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-dw428\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:18.501044 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.500813    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-dw428\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:18.501044 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.500867    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-dw428\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:18.501044 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.500956    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-dw428\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:18.501044 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.500984    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-dw428\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:18.501044 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.501019    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-dw428\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:18.501284 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.501060    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkgw2\" (UniqueName: \"kubernetes.io/projected/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-kube-api-access-tkgw2\") pod \"router-gateway-1-openshift-default-799f46c59b-dw428\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:18.601748 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.601666    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-dw428\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:18.601748 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.601751    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-dw428\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:18.602053 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.601776    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-dw428\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:18.602053 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.601810    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-dw428\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:18.602053 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.601835    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-dw428\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:18.602053 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.601884    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-dw428\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:18.602053 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.601901    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-dw428\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:18.602053 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.602048    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-dw428\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:18.602373 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.602097    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkgw2\" (UniqueName: \"kubernetes.io/projected/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-kube-api-access-tkgw2\") pod \"router-gateway-1-openshift-default-799f46c59b-dw428\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:18.602373 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.602224    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-workload-certs\") pod \"router-gateway-1-openshift-default-799f46c59b-dw428\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:18.602373 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.602284    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-credential-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-dw428\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:18.602543 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.602407    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istio-data\") pod \"router-gateway-1-openshift-default-799f46c59b-dw428\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:18.602543 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.602473    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-workload-socket\") pod \"router-gateway-1-openshift-default-799f46c59b-dw428\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:18.602708 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.602665    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-799f46c59b-dw428\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:18.604648 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.604617    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istio-envoy\") pod \"router-gateway-1-openshift-default-799f46c59b-dw428\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:18.604856 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.604797    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istio-podinfo\") pod \"router-gateway-1-openshift-default-799f46c59b-dw428\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:18.612431 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.612323    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istio-token\") pod \"router-gateway-1-openshift-default-799f46c59b-dw428\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:18.612606 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.612463    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkgw2\" (UniqueName: \"kubernetes.io/projected/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-kube-api-access-tkgw2\") pod \"router-gateway-1-openshift-default-799f46c59b-dw428\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:18.681308 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.681257    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:18.876892 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.876850    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"]
Jul 08 16:46:18.878359 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:46:18.878315    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod899bc306_49c0_45a4_ac99_8d2dd96e8ba5.slice/crio-b55529c99b2c7384acf15e063fd5cec8a893ef4b0afcbf7d3ab17bb4b5a30633 WatchSource:0}: Error finding container b55529c99b2c7384acf15e063fd5cec8a893ef4b0afcbf7d3ab17bb4b5a30633: Status 404 returned error can't find the container with id b55529c99b2c7384acf15e063fd5cec8a893ef4b0afcbf7d3ab17bb4b5a30633
Jul 08 16:46:18.882184 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.882149    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 08 16:46:18.882322 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.882268    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 08 16:46:18.882322 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:18.882314    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 08 16:46:19.699244 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:19.699157    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428" event={"ID":"899bc306-49c0-45a4-ac99-8d2dd96e8ba5","Type":"ContainerStarted","Data":"2fcb8b4ec30177d047d19e1c0cfc0d2c91cd9c9643191683d6c1ca1a63886322"}
Jul 08 16:46:19.700031 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:19.699290    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428" event={"ID":"899bc306-49c0-45a4-ac99-8d2dd96e8ba5","Type":"ContainerStarted","Data":"b55529c99b2c7384acf15e063fd5cec8a893ef4b0afcbf7d3ab17bb4b5a30633"}
Jul 08 16:46:19.734306 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:19.734155    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428" podStartSLOduration=1.7341032159999998 podStartE2EDuration="1.734103216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:46:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:46:19.731820287 +0000 UTC m=+716.445181736" watchObservedRunningTime="2026-07-08 16:46:19.734103216 +0000 UTC m=+716.447464642"
Jul 08 16:46:20.682018 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:20.681946    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:20.692355 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:20.692298    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:20.707356 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:20.707303    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:20.709623 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:20.709561    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:30.585995 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:30.585931    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:32.804267 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:32.804201    2565 generic.go:356] "Generic (PLEG): container finished" podID="899bc306-49c0-45a4-ac99-8d2dd96e8ba5" containerID="2fcb8b4ec30177d047d19e1c0cfc0d2c91cd9c9643191683d6c1ca1a63886322" exitCode=0
Jul 08 16:46:32.804957 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:32.804296    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428" event={"ID":"899bc306-49c0-45a4-ac99-8d2dd96e8ba5","Type":"ContainerDied","Data":"2fcb8b4ec30177d047d19e1c0cfc0d2c91cd9c9643191683d6c1ca1a63886322"}
Jul 08 16:46:32.805049 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:32.805011    2565 scope.go:117] "RemoveContainer" containerID="2fcb8b4ec30177d047d19e1c0cfc0d2c91cd9c9643191683d6c1ca1a63886322"
Jul 08 16:46:32.806513 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:32.806472    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 08 16:46:32.806710 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:32.806571    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 08 16:46:32.806710 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:32.806616    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 08 16:46:33.813043 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:33.812994    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428" event={"ID":"899bc306-49c0-45a4-ac99-8d2dd96e8ba5","Type":"ContainerStarted","Data":"492ace81f53737db1af7d1e2be1656bd44dbe063701e36a6d17bb30394dedc00"}
Jul 08 16:46:34.681820 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:34.681747    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:34.683688 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:34.683628    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428" podUID="899bc306-49c0-45a4-ac99-8d2dd96e8ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 08 16:46:35.682027 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:35.681975    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428" podUID="899bc306-49c0-45a4-ac99-8d2dd96e8ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 08 16:46:35.707164 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:35.707126    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:36.187260 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.187189    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"]
Jul 08 16:46:36.191946 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.191887    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"
Jul 08 16:46:36.195261 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.195214    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qxdr\""
Jul 08 16:46:36.195590 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.195275    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"precise-prefix-cache-test-kserve-self-signed-certs\""
Jul 08 16:46:36.195590 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.195214    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 16:46:36.207606 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.207572    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"]
Jul 08 16:46:36.227148 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.227101    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:36.302828 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.302779    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-tmp-dir\") pod \"precise-prefix-cache-test-kserve-595cd9479-9fl82\" (UID: \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"
Jul 08 16:46:36.302828 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.302840    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-model-cache\") pod \"precise-prefix-cache-test-kserve-595cd9479-9fl82\" (UID: \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"
Jul 08 16:46:36.303099 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.302874    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-dshm\") pod \"precise-prefix-cache-test-kserve-595cd9479-9fl82\" (UID: \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"
Jul 08 16:46:36.303099 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.302911    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-home\") pod \"precise-prefix-cache-test-kserve-595cd9479-9fl82\" (UID: \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"
Jul 08 16:46:36.303099 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.302954    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-595cd9479-9fl82\" (UID: \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"
Jul 08 16:46:36.303099 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.302977    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-tls-certs\") pod \"precise-prefix-cache-test-kserve-595cd9479-9fl82\" (UID: \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"
Jul 08 16:46:36.303099 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.303012    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkkh7\" (UniqueName: \"kubernetes.io/projected/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-kube-api-access-gkkh7\") pod \"precise-prefix-cache-test-kserve-595cd9479-9fl82\" (UID: \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"
Jul 08 16:46:36.404630 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.404546    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-595cd9479-9fl82\" (UID: \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"
Jul 08 16:46:36.404630 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.404619    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-tls-certs\") pod \"precise-prefix-cache-test-kserve-595cd9479-9fl82\" (UID: \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"
Jul 08 16:46:36.405072 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.404668    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkkh7\" (UniqueName: \"kubernetes.io/projected/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-kube-api-access-gkkh7\") pod \"precise-prefix-cache-test-kserve-595cd9479-9fl82\" (UID: \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"
Jul 08 16:46:36.405072 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.404773    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-tmp-dir\") pod \"precise-prefix-cache-test-kserve-595cd9479-9fl82\" (UID: \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"
Jul 08 16:46:36.405072 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.404821    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-model-cache\") pod \"precise-prefix-cache-test-kserve-595cd9479-9fl82\" (UID: \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"
Jul 08 16:46:36.405072 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.404885    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-dshm\") pod \"precise-prefix-cache-test-kserve-595cd9479-9fl82\" (UID: \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"
Jul 08 16:46:36.405072 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.404921    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-home\") pod \"precise-prefix-cache-test-kserve-595cd9479-9fl82\" (UID: \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"
Jul 08 16:46:36.405324 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.405241    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-tmp-dir\") pod \"precise-prefix-cache-test-kserve-595cd9479-9fl82\" (UID: \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"
Jul 08 16:46:36.405324 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.405243    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-kserve-provision-location\") pod \"precise-prefix-cache-test-kserve-595cd9479-9fl82\" (UID: \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"
Jul 08 16:46:36.405324 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.405292    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-model-cache\") pod \"precise-prefix-cache-test-kserve-595cd9479-9fl82\" (UID: \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"
Jul 08 16:46:36.405495 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.405470    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-home\") pod \"precise-prefix-cache-test-kserve-595cd9479-9fl82\" (UID: \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"
Jul 08 16:46:36.407995 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.407958    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-dshm\") pod \"precise-prefix-cache-test-kserve-595cd9479-9fl82\" (UID: \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"
Jul 08 16:46:36.408408 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.408382    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-tls-certs\") pod \"precise-prefix-cache-test-kserve-595cd9479-9fl82\" (UID: \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"
Jul 08 16:46:36.415478 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.415431    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkkh7\" (UniqueName: \"kubernetes.io/projected/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-kube-api-access-gkkh7\") pod \"precise-prefix-cache-test-kserve-595cd9479-9fl82\" (UID: \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\") " pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"
Jul 08 16:46:36.508660 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.508521    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"
Jul 08 16:46:36.586728 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.586636    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:36.682391 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.682320    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428" podUID="899bc306-49c0-45a4-ac99-8d2dd96e8ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 08 16:46:36.754710 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.749290    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"]
Jul 08 16:46:36.762895 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:46:36.762848    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a2181c_9b3f_4aad_8eb3_7b2c93446dec.slice/crio-e61e35c0b7bfea5402c1b10b7cf456789fc6a11865177d99ed3f93259bf17247 WatchSource:0}: Error finding container e61e35c0b7bfea5402c1b10b7cf456789fc6a11865177d99ed3f93259bf17247: Status 404 returned error can't find the container with id e61e35c0b7bfea5402c1b10b7cf456789fc6a11865177d99ed3f93259bf17247
Jul 08 16:46:36.849239 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:36.849084    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82" event={"ID":"48a2181c-9b3f-4aad-8eb3-7b2c93446dec","Type":"ContainerStarted","Data":"e61e35c0b7bfea5402c1b10b7cf456789fc6a11865177d99ed3f93259bf17247"}
Jul 08 16:46:37.682560 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:37.682493    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428" podUID="899bc306-49c0-45a4-ac99-8d2dd96e8ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 08 16:46:38.682179 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:38.682120    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428" podUID="899bc306-49c0-45a4-ac99-8d2dd96e8ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 08 16:46:39.513051 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:39.513001    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:40.008588 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:40.008521    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:40.010755 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:40.010619    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:46:41.900138 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:41.900084    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82" event={"ID":"48a2181c-9b3f-4aad-8eb3-7b2c93446dec","Type":"ContainerStarted","Data":"08459cd4d7ca0dd2786253e5fba582e4a0f3109e012cce28d2872061c4f80bc7"}
Jul 08 16:46:46.934867 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:46.934794    2565 generic.go:356] "Generic (PLEG): container finished" podID="48a2181c-9b3f-4aad-8eb3-7b2c93446dec" containerID="08459cd4d7ca0dd2786253e5fba582e4a0f3109e012cce28d2872061c4f80bc7" exitCode=0
Jul 08 16:46:46.935559 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:46.934873    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82" event={"ID":"48a2181c-9b3f-4aad-8eb3-7b2c93446dec","Type":"ContainerDied","Data":"08459cd4d7ca0dd2786253e5fba582e4a0f3109e012cce28d2872061c4f80bc7"}
Jul 08 16:46:47.389521 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:47.389330    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:47.471794 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:47.471726    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:48.971321 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:48.971241    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82" event={"ID":"48a2181c-9b3f-4aad-8eb3-7b2c93446dec","Type":"ContainerStarted","Data":"5f68bc6b94f5abd1ba69b0bd92574c2238fb9dcb81188f6793710186c78c3b94"}
Jul 08 16:46:48.998861 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:48.998453    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82" podStartSLOduration=2.748781597 podStartE2EDuration="12.998409634s" totalImagesPullingTime="5.659198306s" totalInitContainerRuntime="4.590429731s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:46:36 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:46:48.996412885 +0000 UTC m=+745.709774313" watchObservedRunningTime="2026-07-08 16:46:48.998409634 +0000 UTC m=+745.711771061"
Jul 08 16:46:49.509154 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:49.509068    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"
Jul 08 16:46:52.710963 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:52.710924    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:46:55.371209 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.371170    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"]
Jul 08 16:46:55.420668 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.420620    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"]
Jul 08 16:46:55.420880 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.420772    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:55.550819 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.550775    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwnpw\" (UniqueName: \"kubernetes.io/projected/279f6cb5-9686-4583-b06c-bd27b344f0a9-kube-api-access-bwnpw\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rxfpz\" (UID: \"279f6cb5-9686-4583-b06c-bd27b344f0a9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:55.551067 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.550845    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/279f6cb5-9686-4583-b06c-bd27b344f0a9-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rxfpz\" (UID: \"279f6cb5-9686-4583-b06c-bd27b344f0a9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:55.551067 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.550882    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/279f6cb5-9686-4583-b06c-bd27b344f0a9-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rxfpz\" (UID: \"279f6cb5-9686-4583-b06c-bd27b344f0a9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:55.551067 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.550917    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/279f6cb5-9686-4583-b06c-bd27b344f0a9-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rxfpz\" (UID: \"279f6cb5-9686-4583-b06c-bd27b344f0a9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:55.551067 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.550977    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/279f6cb5-9686-4583-b06c-bd27b344f0a9-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rxfpz\" (UID: \"279f6cb5-9686-4583-b06c-bd27b344f0a9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:55.551259 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.551086    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/279f6cb5-9686-4583-b06c-bd27b344f0a9-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rxfpz\" (UID: \"279f6cb5-9686-4583-b06c-bd27b344f0a9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:55.551259 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.551110    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/279f6cb5-9686-4583-b06c-bd27b344f0a9-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rxfpz\" (UID: \"279f6cb5-9686-4583-b06c-bd27b344f0a9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:55.551259 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.551155    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/279f6cb5-9686-4583-b06c-bd27b344f0a9-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rxfpz\" (UID: \"279f6cb5-9686-4583-b06c-bd27b344f0a9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:55.551259 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.551197    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/279f6cb5-9686-4583-b06c-bd27b344f0a9-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rxfpz\" (UID: \"279f6cb5-9686-4583-b06c-bd27b344f0a9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:55.651922 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.651807    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/279f6cb5-9686-4583-b06c-bd27b344f0a9-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rxfpz\" (UID: \"279f6cb5-9686-4583-b06c-bd27b344f0a9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:55.652130 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.651987    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/279f6cb5-9686-4583-b06c-bd27b344f0a9-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rxfpz\" (UID: \"279f6cb5-9686-4583-b06c-bd27b344f0a9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:55.652130 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.652023    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/279f6cb5-9686-4583-b06c-bd27b344f0a9-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rxfpz\" (UID: \"279f6cb5-9686-4583-b06c-bd27b344f0a9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:55.652130 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.652048    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/279f6cb5-9686-4583-b06c-bd27b344f0a9-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rxfpz\" (UID: \"279f6cb5-9686-4583-b06c-bd27b344f0a9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:55.652130 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.652073    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/279f6cb5-9686-4583-b06c-bd27b344f0a9-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rxfpz\" (UID: \"279f6cb5-9686-4583-b06c-bd27b344f0a9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:55.652281 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.652135    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwnpw\" (UniqueName: \"kubernetes.io/projected/279f6cb5-9686-4583-b06c-bd27b344f0a9-kube-api-access-bwnpw\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rxfpz\" (UID: \"279f6cb5-9686-4583-b06c-bd27b344f0a9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:55.652281 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.652186    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/279f6cb5-9686-4583-b06c-bd27b344f0a9-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rxfpz\" (UID: \"279f6cb5-9686-4583-b06c-bd27b344f0a9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:55.652281 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.652224    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/279f6cb5-9686-4583-b06c-bd27b344f0a9-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rxfpz\" (UID: \"279f6cb5-9686-4583-b06c-bd27b344f0a9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:55.652281 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.652270    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/279f6cb5-9686-4583-b06c-bd27b344f0a9-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rxfpz\" (UID: \"279f6cb5-9686-4583-b06c-bd27b344f0a9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:55.652537 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.652282    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/279f6cb5-9686-4583-b06c-bd27b344f0a9-credential-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rxfpz\" (UID: \"279f6cb5-9686-4583-b06c-bd27b344f0a9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:55.652537 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.652408    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/279f6cb5-9686-4583-b06c-bd27b344f0a9-workload-socket\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rxfpz\" (UID: \"279f6cb5-9686-4583-b06c-bd27b344f0a9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:55.652667 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.652648    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/279f6cb5-9686-4583-b06c-bd27b344f0a9-istio-data\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rxfpz\" (UID: \"279f6cb5-9686-4583-b06c-bd27b344f0a9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:55.653694 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.653630    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/279f6cb5-9686-4583-b06c-bd27b344f0a9-workload-certs\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rxfpz\" (UID: \"279f6cb5-9686-4583-b06c-bd27b344f0a9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:55.654170 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.654138    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/279f6cb5-9686-4583-b06c-bd27b344f0a9-istiod-ca-cert\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rxfpz\" (UID: \"279f6cb5-9686-4583-b06c-bd27b344f0a9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:55.654966 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.654945    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/279f6cb5-9686-4583-b06c-bd27b344f0a9-istio-envoy\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rxfpz\" (UID: \"279f6cb5-9686-4583-b06c-bd27b344f0a9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:55.655649 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.655631    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/279f6cb5-9686-4583-b06c-bd27b344f0a9-istio-podinfo\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rxfpz\" (UID: \"279f6cb5-9686-4583-b06c-bd27b344f0a9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:55.667441 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.667406    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/279f6cb5-9686-4583-b06c-bd27b344f0a9-istio-token\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rxfpz\" (UID: \"279f6cb5-9686-4583-b06c-bd27b344f0a9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:55.667774 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.667749    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwnpw\" (UniqueName: \"kubernetes.io/projected/279f6cb5-9686-4583-b06c-bd27b344f0a9-kube-api-access-bwnpw\") pod \"router-gateway-1-openshift-default-54c47cdfc7-rxfpz\" (UID: \"279f6cb5-9686-4583-b06c-bd27b344f0a9\") " pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:55.733203 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.733155    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:55.953491 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.953446    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"]
Jul 08 16:46:55.955579 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:46:55.955542    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod279f6cb5_9686_4583_b06c_bd27b344f0a9.slice/crio-5a57d7003dc8ba18f1322e65b47cfd3392f2ab0c4397e8382f9815b4c5355e16 WatchSource:0}: Error finding container 5a57d7003dc8ba18f1322e65b47cfd3392f2ab0c4397e8382f9815b4c5355e16: Status 404 returned error can't find the container with id 5a57d7003dc8ba18f1322e65b47cfd3392f2ab0c4397e8382f9815b4c5355e16
Jul 08 16:46:55.958279 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.958226    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 08 16:46:55.958449 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.958359    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 08 16:46:55.958449 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:55.958410    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 08 16:46:56.025770 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:56.025718    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz" event={"ID":"279f6cb5-9686-4583-b06c-bd27b344f0a9","Type":"ContainerStarted","Data":"5a57d7003dc8ba18f1322e65b47cfd3392f2ab0c4397e8382f9815b4c5355e16"}
Jul 08 16:46:56.508958 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:56.508918    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"
Jul 08 16:46:56.522852 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:56.522794    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"
Jul 08 16:46:57.035510 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:57.035435    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz" event={"ID":"279f6cb5-9686-4583-b06c-bd27b344f0a9","Type":"ContainerStarted","Data":"d60f02af33b7ee0dc857f05f26da85d405765c74fe4402efa098edee7e914f1c"}
Jul 08 16:46:57.048085 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:57.048038    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"
Jul 08 16:46:57.066009 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:57.065944    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz" podStartSLOduration=2.065922003 podStartE2EDuration="2.065922003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:46:57.063219158 +0000 UTC m=+753.776580586" watchObservedRunningTime="2026-07-08 16:46:57.065922003 +0000 UTC m=+753.779283431"
Jul 08 16:46:57.733762 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:57.733650    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:57.743148 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:57.743096    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:58.042402 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:58.042279    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:58.043328 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:58.043305    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:46:58.100152 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:58.100081    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"]
Jul 08 16:46:58.100533 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:46:58.100497    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428" podUID="899bc306-49c0-45a4-ac99-8d2dd96e8ba5" containerName="istio-proxy" containerID="cri-o://492ace81f53737db1af7d1e2be1656bd44dbe063701e36a6d17bb30394dedc00" gracePeriod=30
Jul 08 16:47:03.432663 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:03.432592    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:47:03.548554 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:03.548454    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-credential-socket\") pod \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") "
Jul 08 16:47:03.549073 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:03.548661    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-workload-socket\") pod \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") "
Jul 08 16:47:03.549073 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:03.548774    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istio-data\") pod \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") "
Jul 08 16:47:03.549073 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:03.548856    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istio-token\") pod \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") "
Jul 08 16:47:03.549073 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:03.548901    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkgw2\" (UniqueName: \"kubernetes.io/projected/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-kube-api-access-tkgw2\") pod \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") "
Jul 08 16:47:03.549073 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:03.548940    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-workload-certs\") pod \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") "
Jul 08 16:47:03.549073 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:03.548970    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istio-podinfo\") pod \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") "
Jul 08 16:47:03.549073 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:03.549052    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istiod-ca-cert\") pod \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") "
Jul 08 16:47:03.549445 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:03.549129    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istio-envoy\") pod \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\" (UID: \"899bc306-49c0-45a4-ac99-8d2dd96e8ba5\") "
Jul 08 16:47:03.549445 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:03.549169    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "899bc306-49c0-45a4-ac99-8d2dd96e8ba5" (UID: "899bc306-49c0-45a4-ac99-8d2dd96e8ba5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:47:03.549445 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:03.549197    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "899bc306-49c0-45a4-ac99-8d2dd96e8ba5" (UID: "899bc306-49c0-45a4-ac99-8d2dd96e8ba5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:47:03.549445 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:03.549326    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istio-data" (OuterVolumeSpecName: "istio-data") pod "899bc306-49c0-45a4-ac99-8d2dd96e8ba5" (UID: "899bc306-49c0-45a4-ac99-8d2dd96e8ba5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:47:03.549714 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:03.549535    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "899bc306-49c0-45a4-ac99-8d2dd96e8ba5" (UID: "899bc306-49c0-45a4-ac99-8d2dd96e8ba5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:47:03.549811 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:03.549731    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "899bc306-49c0-45a4-ac99-8d2dd96e8ba5" (UID: "899bc306-49c0-45a4-ac99-8d2dd96e8ba5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:47:03.549875 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:03.549844    2565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-credential-socket\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:47:03.549949 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:03.549871    2565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-workload-socket\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:47:03.549949 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:03.549889    2565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istio-data\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:47:03.549949 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:03.549902    2565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-workload-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:47:03.549949 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:03.549913    2565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istiod-ca-cert\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:47:03.553263 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:03.553164    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "899bc306-49c0-45a4-ac99-8d2dd96e8ba5" (UID: "899bc306-49c0-45a4-ac99-8d2dd96e8ba5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:47:03.553263 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:03.553209    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-kube-api-access-tkgw2" (OuterVolumeSpecName: "kube-api-access-tkgw2") pod "899bc306-49c0-45a4-ac99-8d2dd96e8ba5" (UID: "899bc306-49c0-45a4-ac99-8d2dd96e8ba5"). InnerVolumeSpecName "kube-api-access-tkgw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:47:03.553263 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:03.553180    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "899bc306-49c0-45a4-ac99-8d2dd96e8ba5" (UID: "899bc306-49c0-45a4-ac99-8d2dd96e8ba5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 08 16:47:03.553712 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:03.553287    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istio-token" (OuterVolumeSpecName: "istio-token") pod "899bc306-49c0-45a4-ac99-8d2dd96e8ba5" (UID: "899bc306-49c0-45a4-ac99-8d2dd96e8ba5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:47:03.651721 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:03.651450    2565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istio-envoy\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:47:03.651721 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:03.651524    2565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istio-token\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:47:03.651721 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:03.651536    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkgw2\" (UniqueName: \"kubernetes.io/projected/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-kube-api-access-tkgw2\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:47:03.651721 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:03.651548    2565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/899bc306-49c0-45a4-ac99-8d2dd96e8ba5-istio-podinfo\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:47:04.087115 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:04.087074    2565 generic.go:356] "Generic (PLEG): container finished" podID="899bc306-49c0-45a4-ac99-8d2dd96e8ba5" containerID="492ace81f53737db1af7d1e2be1656bd44dbe063701e36a6d17bb30394dedc00" exitCode=0
Jul 08 16:47:04.087344 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:04.087135    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"
Jul 08 16:47:04.087344 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:04.087162    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428" event={"ID":"899bc306-49c0-45a4-ac99-8d2dd96e8ba5","Type":"ContainerDied","Data":"492ace81f53737db1af7d1e2be1656bd44dbe063701e36a6d17bb30394dedc00"}
Jul 08 16:47:04.087344 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:04.087221    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428" event={"ID":"899bc306-49c0-45a4-ac99-8d2dd96e8ba5","Type":"ContainerDied","Data":"b55529c99b2c7384acf15e063fd5cec8a893ef4b0afcbf7d3ab17bb4b5a30633"}
Jul 08 16:47:04.087344 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:04.087245    2565 scope.go:117] "RemoveContainer" containerID="492ace81f53737db1af7d1e2be1656bd44dbe063701e36a6d17bb30394dedc00"
Jul 08 16:47:04.097599 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:04.097578    2565 scope.go:117] "RemoveContainer" containerID="2fcb8b4ec30177d047d19e1c0cfc0d2c91cd9c9643191683d6c1ca1a63886322"
Jul 08 16:47:04.111275 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:04.111229    2565 scope.go:117] "RemoveContainer" containerID="492ace81f53737db1af7d1e2be1656bd44dbe063701e36a6d17bb30394dedc00"
Jul 08 16:47:04.111797 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:47:04.111759    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"492ace81f53737db1af7d1e2be1656bd44dbe063701e36a6d17bb30394dedc00\": container with ID starting with 492ace81f53737db1af7d1e2be1656bd44dbe063701e36a6d17bb30394dedc00 not found: ID does not exist" containerID="492ace81f53737db1af7d1e2be1656bd44dbe063701e36a6d17bb30394dedc00"
Jul 08 16:47:04.111990 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:04.111824    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"492ace81f53737db1af7d1e2be1656bd44dbe063701e36a6d17bb30394dedc00"} err="failed to get container status \"492ace81f53737db1af7d1e2be1656bd44dbe063701e36a6d17bb30394dedc00\": rpc error: code = NotFound desc = could not find container \"492ace81f53737db1af7d1e2be1656bd44dbe063701e36a6d17bb30394dedc00\": container with ID starting with 492ace81f53737db1af7d1e2be1656bd44dbe063701e36a6d17bb30394dedc00 not found: ID does not exist"
Jul 08 16:47:04.111990 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:04.111867    2565 scope.go:117] "RemoveContainer" containerID="2fcb8b4ec30177d047d19e1c0cfc0d2c91cd9c9643191683d6c1ca1a63886322"
Jul 08 16:47:04.112384 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:47:04.112345    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fcb8b4ec30177d047d19e1c0cfc0d2c91cd9c9643191683d6c1ca1a63886322\": container with ID starting with 2fcb8b4ec30177d047d19e1c0cfc0d2c91cd9c9643191683d6c1ca1a63886322 not found: ID does not exist" containerID="2fcb8b4ec30177d047d19e1c0cfc0d2c91cd9c9643191683d6c1ca1a63886322"
Jul 08 16:47:04.112445 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:04.112401    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fcb8b4ec30177d047d19e1c0cfc0d2c91cd9c9643191683d6c1ca1a63886322"} err="failed to get container status \"2fcb8b4ec30177d047d19e1c0cfc0d2c91cd9c9643191683d6c1ca1a63886322\": rpc error: code = NotFound desc = could not find container \"2fcb8b4ec30177d047d19e1c0cfc0d2c91cd9c9643191683d6c1ca1a63886322\": container with ID starting with 2fcb8b4ec30177d047d19e1c0cfc0d2c91cd9c9643191683d6c1ca1a63886322 not found: ID does not exist"
Jul 08 16:47:04.129109 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:04.129026    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"]
Jul 08 16:47:04.135617 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:04.135551    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-gateway-1-openshift-default-799f46c59b-dw428"]
Jul 08 16:47:05.137450 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:05.137405    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:47:05.824805 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:05.824741    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="899bc306-49c0-45a4-ac99-8d2dd96e8ba5" path="/var/lib/kubelet/pods/899bc306-49c0-45a4-ac99-8d2dd96e8ba5/volumes"
Jul 08 16:47:07.244399 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:07.244343    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:47:23.840575 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.840513    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"]
Jul 08 16:47:23.841244 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.841090    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="899bc306-49c0-45a4-ac99-8d2dd96e8ba5" containerName="istio-proxy"
Jul 08 16:47:23.841244 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.841110    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="899bc306-49c0-45a4-ac99-8d2dd96e8ba5" containerName="istio-proxy"
Jul 08 16:47:23.841244 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.841200    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="899bc306-49c0-45a4-ac99-8d2dd96e8ba5" containerName="istio-proxy"
Jul 08 16:47:23.841372 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.841298    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="899bc306-49c0-45a4-ac99-8d2dd96e8ba5" containerName="istio-proxy"
Jul 08 16:47:23.841372 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.841305    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="899bc306-49c0-45a4-ac99-8d2dd96e8ba5" containerName="istio-proxy"
Jul 08 16:47:23.841442 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.841381    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="899bc306-49c0-45a4-ac99-8d2dd96e8ba5" containerName="istio-proxy"
Jul 08 16:47:23.845026 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.844985    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"
Jul 08 16:47:23.848324 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.848287    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-kserve-self-signed-certs\""
Jul 08 16:47:23.863645 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.863588    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"]
Jul 08 16:47:23.868878 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.868819    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-dshm\") pod \"auth-enabled-test-kserve-67f6dd7f6f-p6frc\" (UID: \"c694b852-a4ad-407c-9526-f42f7eecf1fb\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"
Jul 08 16:47:23.869176 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.868963    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c694b852-a4ad-407c-9526-f42f7eecf1fb-tls-certs\") pod \"auth-enabled-test-kserve-67f6dd7f6f-p6frc\" (UID: \"c694b852-a4ad-407c-9526-f42f7eecf1fb\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"
Jul 08 16:47:23.869176 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.869045    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-home\") pod \"auth-enabled-test-kserve-67f6dd7f6f-p6frc\" (UID: \"c694b852-a4ad-407c-9526-f42f7eecf1fb\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"
Jul 08 16:47:23.869309 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.869251    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqwgf\" (UniqueName: \"kubernetes.io/projected/c694b852-a4ad-407c-9526-f42f7eecf1fb-kube-api-access-hqwgf\") pod \"auth-enabled-test-kserve-67f6dd7f6f-p6frc\" (UID: \"c694b852-a4ad-407c-9526-f42f7eecf1fb\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"
Jul 08 16:47:23.869370 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.869331    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-model-cache\") pod \"auth-enabled-test-kserve-67f6dd7f6f-p6frc\" (UID: \"c694b852-a4ad-407c-9526-f42f7eecf1fb\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"
Jul 08 16:47:23.869423 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.869385    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-kserve-provision-location\") pod \"auth-enabled-test-kserve-67f6dd7f6f-p6frc\" (UID: \"c694b852-a4ad-407c-9526-f42f7eecf1fb\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"
Jul 08 16:47:23.869469 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.869410    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-tmp-dir\") pod \"auth-enabled-test-kserve-67f6dd7f6f-p6frc\" (UID: \"c694b852-a4ad-407c-9526-f42f7eecf1fb\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"
Jul 08 16:47:23.872740 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.872693    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:47:23.970762 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.970723    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqwgf\" (UniqueName: \"kubernetes.io/projected/c694b852-a4ad-407c-9526-f42f7eecf1fb-kube-api-access-hqwgf\") pod \"auth-enabled-test-kserve-67f6dd7f6f-p6frc\" (UID: \"c694b852-a4ad-407c-9526-f42f7eecf1fb\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"
Jul 08 16:47:23.970978 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.970774    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-model-cache\") pod \"auth-enabled-test-kserve-67f6dd7f6f-p6frc\" (UID: \"c694b852-a4ad-407c-9526-f42f7eecf1fb\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"
Jul 08 16:47:23.970978 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.970794    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-kserve-provision-location\") pod \"auth-enabled-test-kserve-67f6dd7f6f-p6frc\" (UID: \"c694b852-a4ad-407c-9526-f42f7eecf1fb\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"
Jul 08 16:47:23.970978 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.970816    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-tmp-dir\") pod \"auth-enabled-test-kserve-67f6dd7f6f-p6frc\" (UID: \"c694b852-a4ad-407c-9526-f42f7eecf1fb\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"
Jul 08 16:47:23.970978 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.970855    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-dshm\") pod \"auth-enabled-test-kserve-67f6dd7f6f-p6frc\" (UID: \"c694b852-a4ad-407c-9526-f42f7eecf1fb\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"
Jul 08 16:47:23.970978 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.970885    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c694b852-a4ad-407c-9526-f42f7eecf1fb-tls-certs\") pod \"auth-enabled-test-kserve-67f6dd7f6f-p6frc\" (UID: \"c694b852-a4ad-407c-9526-f42f7eecf1fb\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"
Jul 08 16:47:23.970978 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.970934    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-home\") pod \"auth-enabled-test-kserve-67f6dd7f6f-p6frc\" (UID: \"c694b852-a4ad-407c-9526-f42f7eecf1fb\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"
Jul 08 16:47:23.971300 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.971275    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-model-cache\") pod \"auth-enabled-test-kserve-67f6dd7f6f-p6frc\" (UID: \"c694b852-a4ad-407c-9526-f42f7eecf1fb\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"
Jul 08 16:47:23.971300 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.971295    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-tmp-dir\") pod \"auth-enabled-test-kserve-67f6dd7f6f-p6frc\" (UID: \"c694b852-a4ad-407c-9526-f42f7eecf1fb\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"
Jul 08 16:47:23.971429 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.971357    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-home\") pod \"auth-enabled-test-kserve-67f6dd7f6f-p6frc\" (UID: \"c694b852-a4ad-407c-9526-f42f7eecf1fb\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"
Jul 08 16:47:23.971429 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.971373    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-kserve-provision-location\") pod \"auth-enabled-test-kserve-67f6dd7f6f-p6frc\" (UID: \"c694b852-a4ad-407c-9526-f42f7eecf1fb\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"
Jul 08 16:47:23.973283 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.973260    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-dshm\") pod \"auth-enabled-test-kserve-67f6dd7f6f-p6frc\" (UID: \"c694b852-a4ad-407c-9526-f42f7eecf1fb\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"
Jul 08 16:47:23.973544 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.973528    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c694b852-a4ad-407c-9526-f42f7eecf1fb-tls-certs\") pod \"auth-enabled-test-kserve-67f6dd7f6f-p6frc\" (UID: \"c694b852-a4ad-407c-9526-f42f7eecf1fb\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"
Jul 08 16:47:23.982921 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:23.982891    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqwgf\" (UniqueName: \"kubernetes.io/projected/c694b852-a4ad-407c-9526-f42f7eecf1fb-kube-api-access-hqwgf\") pod \"auth-enabled-test-kserve-67f6dd7f6f-p6frc\" (UID: \"c694b852-a4ad-407c-9526-f42f7eecf1fb\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"
Jul 08 16:47:24.162386 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:24.162154    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"
Jul 08 16:47:24.258719 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:24.258227    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn"]
Jul 08 16:47:24.265515 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:24.265191    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn"
Jul 08 16:47:24.271697 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:24.271059    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-enabled-test-epp-sa-dockercfg-26fcm\""
Jul 08 16:47:24.283727 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:24.283445    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn"]
Jul 08 16:47:24.287218 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:24.287161    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:47:24.376275 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:24.376225    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c76ec0d4-2679-4fb7-8f20-7d1a4723ff17-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn\" (UID: \"c76ec0d4-2679-4fb7-8f20-7d1a4723ff17\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn"
Jul 08 16:47:24.376275 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:24.376273    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpftl\" (UniqueName: \"kubernetes.io/projected/c76ec0d4-2679-4fb7-8f20-7d1a4723ff17-kube-api-access-vpftl\") pod \"auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn\" (UID: \"c76ec0d4-2679-4fb7-8f20-7d1a4723ff17\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn"
Jul 08 16:47:24.376275 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:24.376308    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c76ec0d4-2679-4fb7-8f20-7d1a4723ff17-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn\" (UID: \"c76ec0d4-2679-4fb7-8f20-7d1a4723ff17\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn"
Jul 08 16:47:24.376842 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:24.376343    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c76ec0d4-2679-4fb7-8f20-7d1a4723ff17-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn\" (UID: \"c76ec0d4-2679-4fb7-8f20-7d1a4723ff17\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn"
Jul 08 16:47:24.380976 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:24.380934    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"]
Jul 08 16:47:24.382391 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:47:24.382343    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc694b852_a4ad_407c_9526_f42f7eecf1fb.slice/crio-f3d7fbf52bb2b73e87c546108f86a719baf926394a1816063b2d92a6aa1e5263 WatchSource:0}: Error finding container f3d7fbf52bb2b73e87c546108f86a719baf926394a1816063b2d92a6aa1e5263: Status 404 returned error can't find the container with id f3d7fbf52bb2b73e87c546108f86a719baf926394a1816063b2d92a6aa1e5263
Jul 08 16:47:24.477464 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:24.477411    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c76ec0d4-2679-4fb7-8f20-7d1a4723ff17-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn\" (UID: \"c76ec0d4-2679-4fb7-8f20-7d1a4723ff17\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn"
Jul 08 16:47:24.477731 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:24.477577    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpftl\" (UniqueName: \"kubernetes.io/projected/c76ec0d4-2679-4fb7-8f20-7d1a4723ff17-kube-api-access-vpftl\") pod \"auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn\" (UID: \"c76ec0d4-2679-4fb7-8f20-7d1a4723ff17\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn"
Jul 08 16:47:24.477731 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:24.477632    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c76ec0d4-2679-4fb7-8f20-7d1a4723ff17-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn\" (UID: \"c76ec0d4-2679-4fb7-8f20-7d1a4723ff17\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn"
Jul 08 16:47:24.477731 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:24.477691    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c76ec0d4-2679-4fb7-8f20-7d1a4723ff17-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn\" (UID: \"c76ec0d4-2679-4fb7-8f20-7d1a4723ff17\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn"
Jul 08 16:47:24.477948 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:24.477923    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c76ec0d4-2679-4fb7-8f20-7d1a4723ff17-tokenizer-uds\") pod \"auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn\" (UID: \"c76ec0d4-2679-4fb7-8f20-7d1a4723ff17\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn"
Jul 08 16:47:24.477990 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:24.477948    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c76ec0d4-2679-4fb7-8f20-7d1a4723ff17-kserve-provision-location\") pod \"auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn\" (UID: \"c76ec0d4-2679-4fb7-8f20-7d1a4723ff17\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn"
Jul 08 16:47:24.480775 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:24.480728    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c76ec0d4-2679-4fb7-8f20-7d1a4723ff17-tls-certs\") pod \"auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn\" (UID: \"c76ec0d4-2679-4fb7-8f20-7d1a4723ff17\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn"
Jul 08 16:47:24.488907 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:24.488856    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpftl\" (UniqueName: \"kubernetes.io/projected/c76ec0d4-2679-4fb7-8f20-7d1a4723ff17-kube-api-access-vpftl\") pod \"auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn\" (UID: \"c76ec0d4-2679-4fb7-8f20-7d1a4723ff17\") " pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn"
Jul 08 16:47:24.589459 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:24.589405    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn"
Jul 08 16:47:24.800487 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:24.800429    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn"]
Jul 08 16:47:24.803588 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:47:24.803536    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76ec0d4_2679_4fb7_8f20_7d1a4723ff17.slice/crio-1de18ec3c24fd89470b21902b547f69ae071e332d6ca6795436f3e2a4245ae64 WatchSource:0}: Error finding container 1de18ec3c24fd89470b21902b547f69ae071e332d6ca6795436f3e2a4245ae64: Status 404 returned error can't find the container with id 1de18ec3c24fd89470b21902b547f69ae071e332d6ca6795436f3e2a4245ae64
Jul 08 16:47:25.262708 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:25.261103    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc" event={"ID":"c694b852-a4ad-407c-9526-f42f7eecf1fb","Type":"ContainerStarted","Data":"8ebc82355d27db2045424074ed8c19f7756fa120b21930760e5c969695a351d5"}
Jul 08 16:47:25.262708 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:25.261212    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc" event={"ID":"c694b852-a4ad-407c-9526-f42f7eecf1fb","Type":"ContainerStarted","Data":"f3d7fbf52bb2b73e87c546108f86a719baf926394a1816063b2d92a6aa1e5263"}
Jul 08 16:47:25.267299 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:25.267219    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn" event={"ID":"c76ec0d4-2679-4fb7-8f20-7d1a4723ff17","Type":"ContainerStarted","Data":"d79355978973bdf670dec5d5854c98e55e6d6e50d33130afe9ece9bed1332222"}
Jul 08 16:47:25.267637 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:25.267332    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn" event={"ID":"c76ec0d4-2679-4fb7-8f20-7d1a4723ff17","Type":"ContainerStarted","Data":"1de18ec3c24fd89470b21902b547f69ae071e332d6ca6795436f3e2a4245ae64"}
Jul 08 16:47:26.276496 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:26.276445    2565 generic.go:356] "Generic (PLEG): container finished" podID="c76ec0d4-2679-4fb7-8f20-7d1a4723ff17" containerID="d79355978973bdf670dec5d5854c98e55e6d6e50d33130afe9ece9bed1332222" exitCode=0
Jul 08 16:47:26.276962 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:26.276526    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn" event={"ID":"c76ec0d4-2679-4fb7-8f20-7d1a4723ff17","Type":"ContainerDied","Data":"d79355978973bdf670dec5d5854c98e55e6d6e50d33130afe9ece9bed1332222"}
Jul 08 16:47:28.295515 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:28.295402    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn" event={"ID":"c76ec0d4-2679-4fb7-8f20-7d1a4723ff17","Type":"ContainerStarted","Data":"fb1ba2a8e744696e5f87fb1e0bf6dedf242bab23d13126b383491496c95313fa"}
Jul 08 16:47:28.295984 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:28.295540    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn"
Jul 08 16:47:28.322142 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:28.322059    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn" podStartSLOduration=1.6694649259999998 podStartE2EDuration="4.322030701s" totalImagesPullingTime="1.645682205s" totalInitContainerRuntime="1.00688357s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:47:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:47:28.319333144 +0000 UTC m=+785.032694595" watchObservedRunningTime="2026-07-08 16:47:28.322030701 +0000 UTC m=+785.035392119"
Jul 08 16:47:30.309937 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:30.309888    2565 generic.go:356] "Generic (PLEG): container finished" podID="c694b852-a4ad-407c-9526-f42f7eecf1fb" containerID="8ebc82355d27db2045424074ed8c19f7756fa120b21930760e5c969695a351d5" exitCode=0
Jul 08 16:47:30.310499 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:30.309961    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc" event={"ID":"c694b852-a4ad-407c-9526-f42f7eecf1fb","Type":"ContainerDied","Data":"8ebc82355d27db2045424074ed8c19f7756fa120b21930760e5c969695a351d5"}
Jul 08 16:47:35.053779 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:35.053723    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:47:44.238713 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:44.238642    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:47:44.312515 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:44.312460    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:47:49.684741 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:49.684601    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:47:53.723353 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:53.723303    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"]
Jul 08 16:47:53.724514 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:53.724479    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:47:53.787090 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:53.787030    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:47:59.305513 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:59.305354    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn"
Jul 08 16:47:59.560075 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:59.559872    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc" event={"ID":"c694b852-a4ad-407c-9526-f42f7eecf1fb","Type":"ContainerStarted","Data":"b56641eaf1b46a0104deed0cb5a22b11aecb6025549bc9bcc93dc972e500cbf3"}
Jul 08 16:47:59.586345 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:47:59.586210    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc" podStartSLOduration=2.987206643 podStartE2EDuration="36.586164939s" totalImagesPullingTime="28.715221114s" totalInitContainerRuntime="4.883737182s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:47:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:47:59.584514051 +0000 UTC m=+816.297875508" watchObservedRunningTime="2026-07-08 16:47:59.586164939 +0000 UTC m=+816.299526366"
Jul 08 16:48:00.163821 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:00.163759    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"
Jul 08 16:48:04.163144 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:04.163080    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"
Jul 08 16:48:04.165229 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:04.165180    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc" podUID="c694b852-a4ad-407c-9526-f42f7eecf1fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 08 16:48:08.756203 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:08.756100    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82" podUID="48a2181c-9b3f-4aad-8eb3-7b2c93446dec" containerName="main" containerID="cri-o://5f68bc6b94f5abd1ba69b0bd92574c2238fb9dcb81188f6793710186c78c3b94" gracePeriod=45
Jul 08 16:48:09.037104 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.036900    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82" podUID="48a2181c-9b3f-4aad-8eb3-7b2c93446dec" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": dial tcp 10.133.0.46:8000: connect: connection refused"
Jul 08 16:48:09.614275 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.614224    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"
Jul 08 16:48:09.651699 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.651631    2565 generic.go:356] "Generic (PLEG): container finished" podID="48a2181c-9b3f-4aad-8eb3-7b2c93446dec" containerID="5f68bc6b94f5abd1ba69b0bd92574c2238fb9dcb81188f6793710186c78c3b94" exitCode=0
Jul 08 16:48:09.651958 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.651785    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"
Jul 08 16:48:09.651958 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.651809    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82" event={"ID":"48a2181c-9b3f-4aad-8eb3-7b2c93446dec","Type":"ContainerDied","Data":"5f68bc6b94f5abd1ba69b0bd92574c2238fb9dcb81188f6793710186c78c3b94"}
Jul 08 16:48:09.651958 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.651882    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82" event={"ID":"48a2181c-9b3f-4aad-8eb3-7b2c93446dec","Type":"ContainerDied","Data":"e61e35c0b7bfea5402c1b10b7cf456789fc6a11865177d99ed3f93259bf17247"}
Jul 08 16:48:09.651958 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.651915    2565 scope.go:117] "RemoveContainer" containerID="5f68bc6b94f5abd1ba69b0bd92574c2238fb9dcb81188f6793710186c78c3b94"
Jul 08 16:48:09.665956 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.665917    2565 scope.go:117] "RemoveContainer" containerID="08459cd4d7ca0dd2786253e5fba582e4a0f3109e012cce28d2872061c4f80bc7"
Jul 08 16:48:09.679776 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.679745    2565 scope.go:117] "RemoveContainer" containerID="5f68bc6b94f5abd1ba69b0bd92574c2238fb9dcb81188f6793710186c78c3b94"
Jul 08 16:48:09.680357 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:48:09.680322    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f68bc6b94f5abd1ba69b0bd92574c2238fb9dcb81188f6793710186c78c3b94\": container with ID starting with 5f68bc6b94f5abd1ba69b0bd92574c2238fb9dcb81188f6793710186c78c3b94 not found: ID does not exist" containerID="5f68bc6b94f5abd1ba69b0bd92574c2238fb9dcb81188f6793710186c78c3b94"
Jul 08 16:48:09.680498 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.680375    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f68bc6b94f5abd1ba69b0bd92574c2238fb9dcb81188f6793710186c78c3b94"} err="failed to get container status \"5f68bc6b94f5abd1ba69b0bd92574c2238fb9dcb81188f6793710186c78c3b94\": rpc error: code = NotFound desc = could not find container \"5f68bc6b94f5abd1ba69b0bd92574c2238fb9dcb81188f6793710186c78c3b94\": container with ID starting with 5f68bc6b94f5abd1ba69b0bd92574c2238fb9dcb81188f6793710186c78c3b94 not found: ID does not exist"
Jul 08 16:48:09.680498 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.680405    2565 scope.go:117] "RemoveContainer" containerID="08459cd4d7ca0dd2786253e5fba582e4a0f3109e012cce28d2872061c4f80bc7"
Jul 08 16:48:09.680930 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:48:09.680894    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08459cd4d7ca0dd2786253e5fba582e4a0f3109e012cce28d2872061c4f80bc7\": container with ID starting with 08459cd4d7ca0dd2786253e5fba582e4a0f3109e012cce28d2872061c4f80bc7 not found: ID does not exist" containerID="08459cd4d7ca0dd2786253e5fba582e4a0f3109e012cce28d2872061c4f80bc7"
Jul 08 16:48:09.681012 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.680947    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08459cd4d7ca0dd2786253e5fba582e4a0f3109e012cce28d2872061c4f80bc7"} err="failed to get container status \"08459cd4d7ca0dd2786253e5fba582e4a0f3109e012cce28d2872061c4f80bc7\": rpc error: code = NotFound desc = could not find container \"08459cd4d7ca0dd2786253e5fba582e4a0f3109e012cce28d2872061c4f80bc7\": container with ID starting with 08459cd4d7ca0dd2786253e5fba582e4a0f3109e012cce28d2872061c4f80bc7 not found: ID does not exist"
Jul 08 16:48:09.736988 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.736931    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-kserve-provision-location\") pod \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\" (UID: \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\") "
Jul 08 16:48:09.736988 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.736993    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-tmp-dir\") pod \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\" (UID: \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\") "
Jul 08 16:48:09.737331 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.737032    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkkh7\" (UniqueName: \"kubernetes.io/projected/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-kube-api-access-gkkh7\") pod \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\" (UID: \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\") "
Jul 08 16:48:09.737331 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.737067    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-model-cache\") pod \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\" (UID: \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\") "
Jul 08 16:48:09.737331 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.737102    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-home\") pod \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\" (UID: \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\") "
Jul 08 16:48:09.737331 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.737165    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-dshm\") pod \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\" (UID: \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\") "
Jul 08 16:48:09.737331 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.737220    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-tls-certs\") pod \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\" (UID: \"48a2181c-9b3f-4aad-8eb3-7b2c93446dec\") "
Jul 08 16:48:09.737592 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.737348    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "48a2181c-9b3f-4aad-8eb3-7b2c93446dec" (UID: "48a2181c-9b3f-4aad-8eb3-7b2c93446dec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:48:09.737592 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.737399    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-model-cache" (OuterVolumeSpecName: "model-cache") pod "48a2181c-9b3f-4aad-8eb3-7b2c93446dec" (UID: "48a2181c-9b3f-4aad-8eb3-7b2c93446dec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:48:09.737791 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.737640    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:48:09.737791 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.737658    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:48:09.737951 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.737922    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-home" (OuterVolumeSpecName: "home") pod "48a2181c-9b3f-4aad-8eb3-7b2c93446dec" (UID: "48a2181c-9b3f-4aad-8eb3-7b2c93446dec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:48:09.740695 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.740605    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "48a2181c-9b3f-4aad-8eb3-7b2c93446dec" (UID: "48a2181c-9b3f-4aad-8eb3-7b2c93446dec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:48:09.740695 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.740642    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-kube-api-access-gkkh7" (OuterVolumeSpecName: "kube-api-access-gkkh7") pod "48a2181c-9b3f-4aad-8eb3-7b2c93446dec" (UID: "48a2181c-9b3f-4aad-8eb3-7b2c93446dec"). InnerVolumeSpecName "kube-api-access-gkkh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:48:09.740695 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.740605    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-dshm" (OuterVolumeSpecName: "dshm") pod "48a2181c-9b3f-4aad-8eb3-7b2c93446dec" (UID: "48a2181c-9b3f-4aad-8eb3-7b2c93446dec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:48:09.800430 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.800368    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "48a2181c-9b3f-4aad-8eb3-7b2c93446dec" (UID: "48a2181c-9b3f-4aad-8eb3-7b2c93446dec"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:48:09.838709 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.838650    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:48:09.838709 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.838709    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-kserve-provision-location\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:48:09.838914 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.838721    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkkh7\" (UniqueName: \"kubernetes.io/projected/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-kube-api-access-gkkh7\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:48:09.838914 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.838734    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:48:09.838914 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.838744    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/48a2181c-9b3f-4aad-8eb3-7b2c93446dec-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:48:09.974640 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.974575    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"]
Jul 08 16:48:09.979285 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:09.979203    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/precise-prefix-cache-test-kserve-595cd9479-9fl82"]
Jul 08 16:48:11.824156 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:11.824092    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48a2181c-9b3f-4aad-8eb3-7b2c93446dec" path="/var/lib/kubelet/pods/48a2181c-9b3f-4aad-8eb3-7b2c93446dec/volumes"
Jul 08 16:48:14.163785 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:14.163731    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc" podUID="c694b852-a4ad-407c-9526-f42f7eecf1fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 08 16:48:24.163613 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:24.163524    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc" podUID="c694b852-a4ad-407c-9526-f42f7eecf1fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 08 16:48:28.562992 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:28.562955    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:48:28.966608 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:28.966573    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:48:33.057633 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:33.057597    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:48:34.162753 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:34.162708    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc" podUID="c694b852-a4ad-407c-9526-f42f7eecf1fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 08 16:48:44.162946 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:44.162882    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc" podUID="c694b852-a4ad-407c-9526-f42f7eecf1fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 08 16:48:54.162917 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:48:54.162878    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc" podUID="c694b852-a4ad-407c-9526-f42f7eecf1fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 08 16:49:04.163022 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:49:04.162961    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc" podUID="c694b852-a4ad-407c-9526-f42f7eecf1fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 08 16:49:04.395434 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:49:04.395393    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:49:04.465478 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:49:04.465432    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:49:14.163599 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:49:14.163553    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc" podUID="c694b852-a4ad-407c-9526-f42f7eecf1fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 08 16:49:24.163032 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:49:24.162971    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc" podUID="c694b852-a4ad-407c-9526-f42f7eecf1fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 08 16:49:34.163471 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:49:34.163422    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc" podUID="c694b852-a4ad-407c-9526-f42f7eecf1fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 08 16:49:44.162748 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:49:44.162695    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc" podUID="c694b852-a4ad-407c-9526-f42f7eecf1fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 08 16:49:51.655025 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:49:51.654980    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:49:54.015344 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:49:54.015300    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:49:54.163332 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:49:54.163288    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc" podUID="c694b852-a4ad-407c-9526-f42f7eecf1fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 08 16:49:54.278212 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:49:54.278111    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:50:00.410471 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:00.410406    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:50:04.162943 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:04.162897    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc" podUID="c694b852-a4ad-407c-9526-f42f7eecf1fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 08 16:50:14.174172 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:14.174138    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"
Jul 08 16:50:14.182823 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:14.182790    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"
Jul 08 16:50:19.660558 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:19.660477    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:50:19.662852 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:19.661491    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn"]
Jul 08 16:50:19.666456 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:19.666411    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"]
Jul 08 16:50:19.674734 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:50:19.674661    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 16:50:19.674734 ip-10-0-128-142 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 16:50:19.674734 ip-10-0-128-142 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 08 16:50:19.674734 ip-10-0-128-142 kubenswrapper[2565]:  > containerID="fb1ba2a8e744696e5f87fb1e0bf6dedf242bab23d13126b383491496c95313fa" cmd=["/bin/sleep","15"]
Jul 08 16:50:19.675179 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:19.674771    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn" message="" err=<
Jul 08 16:50:19.675179 ip-10-0-128-142 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 16:50:19.675179 ip-10-0-128-142 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 08 16:50:19.675179 ip-10-0-128-142 kubenswrapper[2565]:  >
Jul 08 16:50:19.675179 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:50:19.674789    2565 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 16:50:19.675179 ip-10-0-128-142 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 16:50:19.675179 ip-10-0-128-142 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 08 16:50:19.675179 ip-10-0-128-142 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn" podUID="c76ec0d4-2679-4fb7-8f20-7d1a4723ff17" containerName="main" containerID="cri-o://fb1ba2a8e744696e5f87fb1e0bf6dedf242bab23d13126b383491496c95313fa"
Jul 08 16:50:19.675179 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:19.674858    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn" podUID="c76ec0d4-2679-4fb7-8f20-7d1a4723ff17" containerName="main" containerID="cri-o://fb1ba2a8e744696e5f87fb1e0bf6dedf242bab23d13126b383491496c95313fa" gracePeriod=60
Jul 08 16:50:19.730720 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:19.730652    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:50:19.955224 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:19.955186    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn"
Jul 08 16:50:20.016262 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:20.016226    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c76ec0d4-2679-4fb7-8f20-7d1a4723ff17-tokenizer-uds\") pod \"c76ec0d4-2679-4fb7-8f20-7d1a4723ff17\" (UID: \"c76ec0d4-2679-4fb7-8f20-7d1a4723ff17\") "
Jul 08 16:50:20.016434 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:20.016299    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c76ec0d4-2679-4fb7-8f20-7d1a4723ff17-kserve-provision-location\") pod \"c76ec0d4-2679-4fb7-8f20-7d1a4723ff17\" (UID: \"c76ec0d4-2679-4fb7-8f20-7d1a4723ff17\") "
Jul 08 16:50:20.016434 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:20.016369    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c76ec0d4-2679-4fb7-8f20-7d1a4723ff17-tls-certs\") pod \"c76ec0d4-2679-4fb7-8f20-7d1a4723ff17\" (UID: \"c76ec0d4-2679-4fb7-8f20-7d1a4723ff17\") "
Jul 08 16:50:20.016434 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:20.016395    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vpftl\" (UniqueName: \"kubernetes.io/projected/c76ec0d4-2679-4fb7-8f20-7d1a4723ff17-kube-api-access-vpftl\") pod \"c76ec0d4-2679-4fb7-8f20-7d1a4723ff17\" (UID: \"c76ec0d4-2679-4fb7-8f20-7d1a4723ff17\") "
Jul 08 16:50:20.016647 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:20.016623    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76ec0d4-2679-4fb7-8f20-7d1a4723ff17-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "c76ec0d4-2679-4fb7-8f20-7d1a4723ff17" (UID: "c76ec0d4-2679-4fb7-8f20-7d1a4723ff17"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:50:20.016794 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:20.016778    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c76ec0d4-2679-4fb7-8f20-7d1a4723ff17-tokenizer-uds\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:50:20.017165 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:20.017139    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c76ec0d4-2679-4fb7-8f20-7d1a4723ff17-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c76ec0d4-2679-4fb7-8f20-7d1a4723ff17" (UID: "c76ec0d4-2679-4fb7-8f20-7d1a4723ff17"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:50:20.018743 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:20.018707    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76ec0d4-2679-4fb7-8f20-7d1a4723ff17-kube-api-access-vpftl" (OuterVolumeSpecName: "kube-api-access-vpftl") pod "c76ec0d4-2679-4fb7-8f20-7d1a4723ff17" (UID: "c76ec0d4-2679-4fb7-8f20-7d1a4723ff17"). InnerVolumeSpecName "kube-api-access-vpftl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:50:20.018743 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:20.018724    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c76ec0d4-2679-4fb7-8f20-7d1a4723ff17-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c76ec0d4-2679-4fb7-8f20-7d1a4723ff17" (UID: "c76ec0d4-2679-4fb7-8f20-7d1a4723ff17"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:50:20.117285 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:20.117233    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c76ec0d4-2679-4fb7-8f20-7d1a4723ff17-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:50:20.117285 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:20.117275    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpftl\" (UniqueName: \"kubernetes.io/projected/c76ec0d4-2679-4fb7-8f20-7d1a4723ff17-kube-api-access-vpftl\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:50:20.117285 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:20.117286    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c76ec0d4-2679-4fb7-8f20-7d1a4723ff17-kserve-provision-location\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:50:20.467048 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:20.466991    2565 generic.go:356] "Generic (PLEG): container finished" podID="c76ec0d4-2679-4fb7-8f20-7d1a4723ff17" containerID="fb1ba2a8e744696e5f87fb1e0bf6dedf242bab23d13126b383491496c95313fa" exitCode=0
Jul 08 16:50:20.467319 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:20.467076    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn"
Jul 08 16:50:20.467319 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:20.467068    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn" event={"ID":"c76ec0d4-2679-4fb7-8f20-7d1a4723ff17","Type":"ContainerDied","Data":"fb1ba2a8e744696e5f87fb1e0bf6dedf242bab23d13126b383491496c95313fa"}
Jul 08 16:50:20.467319 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:20.467223    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn" event={"ID":"c76ec0d4-2679-4fb7-8f20-7d1a4723ff17","Type":"ContainerDied","Data":"1de18ec3c24fd89470b21902b547f69ae071e332d6ca6795436f3e2a4245ae64"}
Jul 08 16:50:20.467319 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:20.467242    2565 scope.go:117] "RemoveContainer" containerID="fb1ba2a8e744696e5f87fb1e0bf6dedf242bab23d13126b383491496c95313fa"
Jul 08 16:50:20.479687 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:20.479647    2565 scope.go:117] "RemoveContainer" containerID="d79355978973bdf670dec5d5854c98e55e6d6e50d33130afe9ece9bed1332222"
Jul 08 16:50:20.489108 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:20.489086    2565 scope.go:117] "RemoveContainer" containerID="fb1ba2a8e744696e5f87fb1e0bf6dedf242bab23d13126b383491496c95313fa"
Jul 08 16:50:20.489450 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:50:20.489428    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb1ba2a8e744696e5f87fb1e0bf6dedf242bab23d13126b383491496c95313fa\": container with ID starting with fb1ba2a8e744696e5f87fb1e0bf6dedf242bab23d13126b383491496c95313fa not found: ID does not exist" containerID="fb1ba2a8e744696e5f87fb1e0bf6dedf242bab23d13126b383491496c95313fa"
Jul 08 16:50:20.489540 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:20.489467    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb1ba2a8e744696e5f87fb1e0bf6dedf242bab23d13126b383491496c95313fa"} err="failed to get container status \"fb1ba2a8e744696e5f87fb1e0bf6dedf242bab23d13126b383491496c95313fa\": rpc error: code = NotFound desc = could not find container \"fb1ba2a8e744696e5f87fb1e0bf6dedf242bab23d13126b383491496c95313fa\": container with ID starting with fb1ba2a8e744696e5f87fb1e0bf6dedf242bab23d13126b383491496c95313fa not found: ID does not exist"
Jul 08 16:50:20.489540 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:20.489498    2565 scope.go:117] "RemoveContainer" containerID="d79355978973bdf670dec5d5854c98e55e6d6e50d33130afe9ece9bed1332222"
Jul 08 16:50:20.489864 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:50:20.489836    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d79355978973bdf670dec5d5854c98e55e6d6e50d33130afe9ece9bed1332222\": container with ID starting with d79355978973bdf670dec5d5854c98e55e6d6e50d33130afe9ece9bed1332222 not found: ID does not exist" containerID="d79355978973bdf670dec5d5854c98e55e6d6e50d33130afe9ece9bed1332222"
Jul 08 16:50:20.489937 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:20.489873    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d79355978973bdf670dec5d5854c98e55e6d6e50d33130afe9ece9bed1332222"} err="failed to get container status \"d79355978973bdf670dec5d5854c98e55e6d6e50d33130afe9ece9bed1332222\": rpc error: code = NotFound desc = could not find container \"d79355978973bdf670dec5d5854c98e55e6d6e50d33130afe9ece9bed1332222\": container with ID starting with d79355978973bdf670dec5d5854c98e55e6d6e50d33130afe9ece9bed1332222 not found: ID does not exist"
Jul 08 16:50:20.494689 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:20.494636    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn"]
Jul 08 16:50:20.498385 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:20.498336    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-router-scheduler-857b4f747c-7k9fn"]
Jul 08 16:50:21.822459 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:21.822425    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c76ec0d4-2679-4fb7-8f20-7d1a4723ff17" path="/var/lib/kubelet/pods/c76ec0d4-2679-4fb7-8f20-7d1a4723ff17/volumes"
Jul 08 16:50:34.696164 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:34.696109    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc" podUID="c694b852-a4ad-407c-9526-f42f7eecf1fb" containerName="main" containerID="cri-o://b56641eaf1b46a0104deed0cb5a22b11aecb6025549bc9bcc93dc972e500cbf3" gracePeriod=45
Jul 08 16:50:35.175011 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.174965    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc" podUID="c694b852-a4ad-407c-9526-f42f7eecf1fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": dial tcp 10.133.0.48:8000: connect: connection refused"
Jul 08 16:50:35.461398 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.461368    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"
Jul 08 16:50:35.553912 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.553877    2565 generic.go:356] "Generic (PLEG): container finished" podID="c694b852-a4ad-407c-9526-f42f7eecf1fb" containerID="b56641eaf1b46a0104deed0cb5a22b11aecb6025549bc9bcc93dc972e500cbf3" exitCode=0
Jul 08 16:50:35.554087 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.553925    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"
Jul 08 16:50:35.554087 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.553948    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc" event={"ID":"c694b852-a4ad-407c-9526-f42f7eecf1fb","Type":"ContainerDied","Data":"b56641eaf1b46a0104deed0cb5a22b11aecb6025549bc9bcc93dc972e500cbf3"}
Jul 08 16:50:35.554087 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.554004    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc" event={"ID":"c694b852-a4ad-407c-9526-f42f7eecf1fb","Type":"ContainerDied","Data":"f3d7fbf52bb2b73e87c546108f86a719baf926394a1816063b2d92a6aa1e5263"}
Jul 08 16:50:35.554087 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.554026    2565 scope.go:117] "RemoveContainer" containerID="b56641eaf1b46a0104deed0cb5a22b11aecb6025549bc9bcc93dc972e500cbf3"
Jul 08 16:50:35.563056 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.563036    2565 scope.go:117] "RemoveContainer" containerID="8ebc82355d27db2045424074ed8c19f7756fa120b21930760e5c969695a351d5"
Jul 08 16:50:35.569443 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.569409    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-dshm\") pod \"c694b852-a4ad-407c-9526-f42f7eecf1fb\" (UID: \"c694b852-a4ad-407c-9526-f42f7eecf1fb\") "
Jul 08 16:50:35.569602 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.569468    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c694b852-a4ad-407c-9526-f42f7eecf1fb-tls-certs\") pod \"c694b852-a4ad-407c-9526-f42f7eecf1fb\" (UID: \"c694b852-a4ad-407c-9526-f42f7eecf1fb\") "
Jul 08 16:50:35.569602 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.569528    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-home\") pod \"c694b852-a4ad-407c-9526-f42f7eecf1fb\" (UID: \"c694b852-a4ad-407c-9526-f42f7eecf1fb\") "
Jul 08 16:50:35.569602 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.569560    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hqwgf\" (UniqueName: \"kubernetes.io/projected/c694b852-a4ad-407c-9526-f42f7eecf1fb-kube-api-access-hqwgf\") pod \"c694b852-a4ad-407c-9526-f42f7eecf1fb\" (UID: \"c694b852-a4ad-407c-9526-f42f7eecf1fb\") "
Jul 08 16:50:35.569788 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.569623    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-tmp-dir\") pod \"c694b852-a4ad-407c-9526-f42f7eecf1fb\" (UID: \"c694b852-a4ad-407c-9526-f42f7eecf1fb\") "
Jul 08 16:50:35.569788 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.569704    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-kserve-provision-location\") pod \"c694b852-a4ad-407c-9526-f42f7eecf1fb\" (UID: \"c694b852-a4ad-407c-9526-f42f7eecf1fb\") "
Jul 08 16:50:35.569788 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.569734    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-model-cache\") pod \"c694b852-a4ad-407c-9526-f42f7eecf1fb\" (UID: \"c694b852-a4ad-407c-9526-f42f7eecf1fb\") "
Jul 08 16:50:35.570224 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.570194    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-model-cache" (OuterVolumeSpecName: "model-cache") pod "c694b852-a4ad-407c-9526-f42f7eecf1fb" (UID: "c694b852-a4ad-407c-9526-f42f7eecf1fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:50:35.570328 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.570263    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-home" (OuterVolumeSpecName: "home") pod "c694b852-a4ad-407c-9526-f42f7eecf1fb" (UID: "c694b852-a4ad-407c-9526-f42f7eecf1fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:50:35.572181 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.572152    2565 scope.go:117] "RemoveContainer" containerID="b56641eaf1b46a0104deed0cb5a22b11aecb6025549bc9bcc93dc972e500cbf3"
Jul 08 16:50:35.572400 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.572377    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c694b852-a4ad-407c-9526-f42f7eecf1fb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c694b852-a4ad-407c-9526-f42f7eecf1fb" (UID: "c694b852-a4ad-407c-9526-f42f7eecf1fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:50:35.572521 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.572497    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-dshm" (OuterVolumeSpecName: "dshm") pod "c694b852-a4ad-407c-9526-f42f7eecf1fb" (UID: "c694b852-a4ad-407c-9526-f42f7eecf1fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:50:35.572580 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:50:35.572516    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b56641eaf1b46a0104deed0cb5a22b11aecb6025549bc9bcc93dc972e500cbf3\": container with ID starting with b56641eaf1b46a0104deed0cb5a22b11aecb6025549bc9bcc93dc972e500cbf3 not found: ID does not exist" containerID="b56641eaf1b46a0104deed0cb5a22b11aecb6025549bc9bcc93dc972e500cbf3"
Jul 08 16:50:35.572580 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.572538    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c694b852-a4ad-407c-9526-f42f7eecf1fb-kube-api-access-hqwgf" (OuterVolumeSpecName: "kube-api-access-hqwgf") pod "c694b852-a4ad-407c-9526-f42f7eecf1fb" (UID: "c694b852-a4ad-407c-9526-f42f7eecf1fb"). InnerVolumeSpecName "kube-api-access-hqwgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:50:35.572580 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.572553    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56641eaf1b46a0104deed0cb5a22b11aecb6025549bc9bcc93dc972e500cbf3"} err="failed to get container status \"b56641eaf1b46a0104deed0cb5a22b11aecb6025549bc9bcc93dc972e500cbf3\": rpc error: code = NotFound desc = could not find container \"b56641eaf1b46a0104deed0cb5a22b11aecb6025549bc9bcc93dc972e500cbf3\": container with ID starting with b56641eaf1b46a0104deed0cb5a22b11aecb6025549bc9bcc93dc972e500cbf3 not found: ID does not exist"
Jul 08 16:50:35.572580 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.572578    2565 scope.go:117] "RemoveContainer" containerID="8ebc82355d27db2045424074ed8c19f7756fa120b21930760e5c969695a351d5"
Jul 08 16:50:35.572892 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:50:35.572874    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ebc82355d27db2045424074ed8c19f7756fa120b21930760e5c969695a351d5\": container with ID starting with 8ebc82355d27db2045424074ed8c19f7756fa120b21930760e5c969695a351d5 not found: ID does not exist" containerID="8ebc82355d27db2045424074ed8c19f7756fa120b21930760e5c969695a351d5"
Jul 08 16:50:35.572948 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.572898    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ebc82355d27db2045424074ed8c19f7756fa120b21930760e5c969695a351d5"} err="failed to get container status \"8ebc82355d27db2045424074ed8c19f7756fa120b21930760e5c969695a351d5\": rpc error: code = NotFound desc = could not find container \"8ebc82355d27db2045424074ed8c19f7756fa120b21930760e5c969695a351d5\": container with ID starting with 8ebc82355d27db2045424074ed8c19f7756fa120b21930760e5c969695a351d5 not found: ID does not exist"
Jul 08 16:50:35.586772 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.586744    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c694b852-a4ad-407c-9526-f42f7eecf1fb" (UID: "c694b852-a4ad-407c-9526-f42f7eecf1fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:50:35.630363 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.630303    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c694b852-a4ad-407c-9526-f42f7eecf1fb" (UID: "c694b852-a4ad-407c-9526-f42f7eecf1fb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:50:35.671222 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.671187    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-kserve-provision-location\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:50:35.671222 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.671219    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:50:35.671222 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.671229    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:50:35.671492 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.671237    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c694b852-a4ad-407c-9526-f42f7eecf1fb-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:50:35.671492 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.671245    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:50:35.671492 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.671253    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqwgf\" (UniqueName: \"kubernetes.io/projected/c694b852-a4ad-407c-9526-f42f7eecf1fb-kube-api-access-hqwgf\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:50:35.671492 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.671261    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c694b852-a4ad-407c-9526-f42f7eecf1fb-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:50:35.875313 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.875280    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"]
Jul 08 16:50:35.879551 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:35.879517    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-enabled-test-kserve-67f6dd7f6f-p6frc"]
Jul 08 16:50:37.821495 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:37.821457    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c694b852-a4ad-407c-9526-f42f7eecf1fb" path="/var/lib/kubelet/pods/c694b852-a4ad-407c-9526-f42f7eecf1fb/volumes"
Jul 08 16:50:41.896117 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:41.896085    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:50:42.208744 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.208711    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8"]
Jul 08 16:50:42.209313 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.209283    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48a2181c-9b3f-4aad-8eb3-7b2c93446dec" containerName="main"
Jul 08 16:50:42.209313 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.209313    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="48a2181c-9b3f-4aad-8eb3-7b2c93446dec" containerName="main"
Jul 08 16:50:42.209555 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.209327    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c76ec0d4-2679-4fb7-8f20-7d1a4723ff17" containerName="storage-initializer"
Jul 08 16:50:42.209555 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.209336    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c76ec0d4-2679-4fb7-8f20-7d1a4723ff17" containerName="storage-initializer"
Jul 08 16:50:42.209555 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.209348    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48a2181c-9b3f-4aad-8eb3-7b2c93446dec" containerName="storage-initializer"
Jul 08 16:50:42.209555 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.209356    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="48a2181c-9b3f-4aad-8eb3-7b2c93446dec" containerName="storage-initializer"
Jul 08 16:50:42.209555 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.209370    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c694b852-a4ad-407c-9526-f42f7eecf1fb" containerName="storage-initializer"
Jul 08 16:50:42.209555 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.209378    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c694b852-a4ad-407c-9526-f42f7eecf1fb" containerName="storage-initializer"
Jul 08 16:50:42.209555 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.209400    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c694b852-a4ad-407c-9526-f42f7eecf1fb" containerName="main"
Jul 08 16:50:42.209555 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.209409    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c694b852-a4ad-407c-9526-f42f7eecf1fb" containerName="main"
Jul 08 16:50:42.209555 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.209420    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c76ec0d4-2679-4fb7-8f20-7d1a4723ff17" containerName="main"
Jul 08 16:50:42.209555 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.209428    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="c76ec0d4-2679-4fb7-8f20-7d1a4723ff17" containerName="main"
Jul 08 16:50:42.209555 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.209534    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c76ec0d4-2679-4fb7-8f20-7d1a4723ff17" containerName="main"
Jul 08 16:50:42.209555 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.209547    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="c694b852-a4ad-407c-9526-f42f7eecf1fb" containerName="main"
Jul 08 16:50:42.209555 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.209559    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="48a2181c-9b3f-4aad-8eb3-7b2c93446dec" containerName="main"
Jul 08 16:50:42.214912 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.214886    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8"
Jul 08 16:50:42.218521 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.218496    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb11a9c9832b99b016bc8f8e0ea095712-kserve-self-signed-certs\""
Jul 08 16:50:42.218521 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.218509    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-a50492e9-epp-sa-dockercfg-wjmbr\""
Jul 08 16:50:42.218758 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.218496    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qxdr\""
Jul 08 16:50:42.224967 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.224941    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8"]
Jul 08 16:50:42.248808 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.248772    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:50:42.333895 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.333849    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8\" (UID: \"55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8"
Jul 08 16:50:42.334094 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.333908    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bzdz\" (UniqueName: \"kubernetes.io/projected/55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90-kube-api-access-5bzdz\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8\" (UID: \"55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8"
Jul 08 16:50:42.334094 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.333983    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8\" (UID: \"55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8"
Jul 08 16:50:42.435257 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.435203    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bzdz\" (UniqueName: \"kubernetes.io/projected/55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90-kube-api-access-5bzdz\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8\" (UID: \"55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8"
Jul 08 16:50:42.435523 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.435296    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8\" (UID: \"55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8"
Jul 08 16:50:42.435523 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.435383    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8\" (UID: \"55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8"
Jul 08 16:50:42.435846 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.435824    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90-tokenizer-uds\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8\" (UID: \"55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8"
Jul 08 16:50:42.438492 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.438460    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90-tls-certs\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8\" (UID: \"55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8"
Jul 08 16:50:42.447153 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.447099    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bzdz\" (UniqueName: \"kubernetes.io/projected/55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90-kube-api-access-5bzdz\") pod \"llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8\" (UID: \"55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90\") " pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8"
Jul 08 16:50:42.526494 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.526378    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8"
Jul 08 16:50:42.676442 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:42.676416    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8"]
Jul 08 16:50:42.678010 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:50:42.677977    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e76bc0_af1b_421f_a4a6_f6dcb3e2ba90.slice/crio-a9011d2e6e47e3da8ec9b0eb7a0c53a8b23e82be5782216251e86da0b68d19fa WatchSource:0}: Error finding container a9011d2e6e47e3da8ec9b0eb7a0c53a8b23e82be5782216251e86da0b68d19fa: Status 404 returned error can't find the container with id a9011d2e6e47e3da8ec9b0eb7a0c53a8b23e82be5782216251e86da0b68d19fa
Jul 08 16:50:43.600199 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:43.600158    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8" event={"ID":"55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90","Type":"ContainerStarted","Data":"8bf99c6a2c2d4d41ac1b1e6dac2c4c13c07288e37e517a930325ef1f96b1a720"}
Jul 08 16:50:43.600603 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:43.600214    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8" event={"ID":"55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90","Type":"ContainerStarted","Data":"a9011d2e6e47e3da8ec9b0eb7a0c53a8b23e82be5782216251e86da0b68d19fa"}
Jul 08 16:50:43.600603 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:43.600257    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8"
Jul 08 16:50:43.620051 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:43.619980    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8" podStartSLOduration=1.61996108 podStartE2EDuration="1.61996108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:50:43.618354726 +0000 UTC m=+980.331716183" watchObservedRunningTime="2026-07-08 16:50:43.61996108 +0000 UTC m=+980.333322505"
Jul 08 16:50:55.291020 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.290881    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"]
Jul 08 16:50:55.295449 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.295378    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"
Jul 08 16:50:55.298221 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.298183    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 16:50:55.298478 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.298244    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-kserve-self-signed-certs\""
Jul 08 16:50:55.304293 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.304263    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"]
Jul 08 16:50:55.325417 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.325379    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:50:55.353575 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.353533    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-69f89b8bf7-whpjz\" (UID: \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"
Jul 08 16:50:55.353575 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.353578    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-tls-certs\") pod \"auth-invalid-token-test-kserve-69f89b8bf7-whpjz\" (UID: \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"
Jul 08 16:50:55.353847 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.353598    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-tmp-dir\") pod \"auth-invalid-token-test-kserve-69f89b8bf7-whpjz\" (UID: \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"
Jul 08 16:50:55.353847 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.353653    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-home\") pod \"auth-invalid-token-test-kserve-69f89b8bf7-whpjz\" (UID: \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"
Jul 08 16:50:55.353941 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.353730    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-model-cache\") pod \"auth-invalid-token-test-kserve-69f89b8bf7-whpjz\" (UID: \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"
Jul 08 16:50:55.356875 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.354262    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mkp6\" (UniqueName: \"kubernetes.io/projected/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-kube-api-access-2mkp6\") pod \"auth-invalid-token-test-kserve-69f89b8bf7-whpjz\" (UID: \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"
Jul 08 16:50:55.356875 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.354322    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-dshm\") pod \"auth-invalid-token-test-kserve-69f89b8bf7-whpjz\" (UID: \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"
Jul 08 16:50:55.454924 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.454878    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-model-cache\") pod \"auth-invalid-token-test-kserve-69f89b8bf7-whpjz\" (UID: \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"
Jul 08 16:50:55.454924 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.454926    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mkp6\" (UniqueName: \"kubernetes.io/projected/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-kube-api-access-2mkp6\") pod \"auth-invalid-token-test-kserve-69f89b8bf7-whpjz\" (UID: \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"
Jul 08 16:50:55.455159 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.455058    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-dshm\") pod \"auth-invalid-token-test-kserve-69f89b8bf7-whpjz\" (UID: \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"
Jul 08 16:50:55.455212 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.455192    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-69f89b8bf7-whpjz\" (UID: \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"
Jul 08 16:50:55.455261 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.455247    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-tls-certs\") pod \"auth-invalid-token-test-kserve-69f89b8bf7-whpjz\" (UID: \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"
Jul 08 16:50:55.455302 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.455285    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-tmp-dir\") pod \"auth-invalid-token-test-kserve-69f89b8bf7-whpjz\" (UID: \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"
Jul 08 16:50:55.455346 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.455326    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-home\") pod \"auth-invalid-token-test-kserve-69f89b8bf7-whpjz\" (UID: \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"
Jul 08 16:50:55.455389 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.455371    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-model-cache\") pod \"auth-invalid-token-test-kserve-69f89b8bf7-whpjz\" (UID: \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"
Jul 08 16:50:55.455544 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.455523    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-69f89b8bf7-whpjz\" (UID: \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"
Jul 08 16:50:55.455743 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.455722    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-tmp-dir\") pod \"auth-invalid-token-test-kserve-69f89b8bf7-whpjz\" (UID: \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"
Jul 08 16:50:55.455743 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.455738    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-home\") pod \"auth-invalid-token-test-kserve-69f89b8bf7-whpjz\" (UID: \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"
Jul 08 16:50:55.457398 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.457371    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-dshm\") pod \"auth-invalid-token-test-kserve-69f89b8bf7-whpjz\" (UID: \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"
Jul 08 16:50:55.457626 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.457609    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-tls-certs\") pod \"auth-invalid-token-test-kserve-69f89b8bf7-whpjz\" (UID: \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"
Jul 08 16:50:55.465367 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.465337    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mkp6\" (UniqueName: \"kubernetes.io/projected/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-kube-api-access-2mkp6\") pod \"auth-invalid-token-test-kserve-69f89b8bf7-whpjz\" (UID: \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"
Jul 08 16:50:55.545331 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.545241    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc"]
Jul 08 16:50:55.549595 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.549565    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc"
Jul 08 16:50:55.552196 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.552170    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-invalid-token-test-epp-sa-dockercfg-4qjtb\""
Jul 08 16:50:55.557087 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.556302    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/457297bb-276a-4629-9b56-e4e6033e9d3b-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc\" (UID: \"457297bb-276a-4629-9b56-e4e6033e9d3b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc"
Jul 08 16:50:55.557087 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.556491    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/457297bb-276a-4629-9b56-e4e6033e9d3b-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc\" (UID: \"457297bb-276a-4629-9b56-e4e6033e9d3b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc"
Jul 08 16:50:55.557326 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.557222    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6zbr\" (UniqueName: \"kubernetes.io/projected/457297bb-276a-4629-9b56-e4e6033e9d3b-kube-api-access-g6zbr\") pod \"auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc\" (UID: \"457297bb-276a-4629-9b56-e4e6033e9d3b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc"
Jul 08 16:50:55.557326 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.557301    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/457297bb-276a-4629-9b56-e4e6033e9d3b-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc\" (UID: \"457297bb-276a-4629-9b56-e4e6033e9d3b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc"
Jul 08 16:50:55.559639 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.559603    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc"]
Jul 08 16:50:55.611660 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.611603    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"
Jul 08 16:50:55.658858 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.658669    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/457297bb-276a-4629-9b56-e4e6033e9d3b-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc\" (UID: \"457297bb-276a-4629-9b56-e4e6033e9d3b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc"
Jul 08 16:50:55.658858 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.658787    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6zbr\" (UniqueName: \"kubernetes.io/projected/457297bb-276a-4629-9b56-e4e6033e9d3b-kube-api-access-g6zbr\") pod \"auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc\" (UID: \"457297bb-276a-4629-9b56-e4e6033e9d3b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc"
Jul 08 16:50:55.658858 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.658850    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/457297bb-276a-4629-9b56-e4e6033e9d3b-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc\" (UID: \"457297bb-276a-4629-9b56-e4e6033e9d3b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc"
Jul 08 16:50:55.659196 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.658902    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/457297bb-276a-4629-9b56-e4e6033e9d3b-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc\" (UID: \"457297bb-276a-4629-9b56-e4e6033e9d3b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc"
Jul 08 16:50:55.659543 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.659370    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/457297bb-276a-4629-9b56-e4e6033e9d3b-kserve-provision-location\") pod \"auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc\" (UID: \"457297bb-276a-4629-9b56-e4e6033e9d3b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc"
Jul 08 16:50:55.659543 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.659415    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/457297bb-276a-4629-9b56-e4e6033e9d3b-tokenizer-uds\") pod \"auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc\" (UID: \"457297bb-276a-4629-9b56-e4e6033e9d3b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc"
Jul 08 16:50:55.662371 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.662323    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/457297bb-276a-4629-9b56-e4e6033e9d3b-tls-certs\") pod \"auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc\" (UID: \"457297bb-276a-4629-9b56-e4e6033e9d3b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc"
Jul 08 16:50:55.670048 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.669995    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6zbr\" (UniqueName: \"kubernetes.io/projected/457297bb-276a-4629-9b56-e4e6033e9d3b-kube-api-access-g6zbr\") pod \"auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc\" (UID: \"457297bb-276a-4629-9b56-e4e6033e9d3b\") " pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc"
Jul 08 16:50:55.774234 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.774190    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"]
Jul 08 16:50:55.775391 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:50:55.775365    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc290a5_d7cb_458d_a328_0e7aa97bb7df.slice/crio-434ecdbb8f60fa4a4472d95291a3485b2817e5f3da1a9889a4d72817a921564a WatchSource:0}: Error finding container 434ecdbb8f60fa4a4472d95291a3485b2817e5f3da1a9889a4d72817a921564a: Status 404 returned error can't find the container with id 434ecdbb8f60fa4a4472d95291a3485b2817e5f3da1a9889a4d72817a921564a
Jul 08 16:50:55.861217 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:55.861171    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc"
Jul 08 16:50:56.015691 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:56.015644    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc"]
Jul 08 16:50:56.016724 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:50:56.016671    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457297bb_276a_4629_9b56_e4e6033e9d3b.slice/crio-c1be136377b9bde399c2614d0efd4fb11e87a22f988adca88c6b8a8f9fb96719 WatchSource:0}: Error finding container c1be136377b9bde399c2614d0efd4fb11e87a22f988adca88c6b8a8f9fb96719: Status 404 returned error can't find the container with id c1be136377b9bde399c2614d0efd4fb11e87a22f988adca88c6b8a8f9fb96719
Jul 08 16:50:56.676970 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:56.676931    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc" event={"ID":"457297bb-276a-4629-9b56-e4e6033e9d3b","Type":"ContainerStarted","Data":"fc8195358770e1cdb75b9b04dab181804d33821f6b6e4b3c67c26ada5e725145"}
Jul 08 16:50:56.677417 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:56.676990    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc" event={"ID":"457297bb-276a-4629-9b56-e4e6033e9d3b","Type":"ContainerStarted","Data":"c1be136377b9bde399c2614d0efd4fb11e87a22f988adca88c6b8a8f9fb96719"}
Jul 08 16:50:56.678724 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:56.678696    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz" event={"ID":"cdc290a5-d7cb-458d-a328-0e7aa97bb7df","Type":"ContainerStarted","Data":"3c6b9906fe33b2b1794371d9c4dcc93fa5e961b017b2221b90b49d2ea16dbaa3"}
Jul 08 16:50:56.678878 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:56.678745    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz" event={"ID":"cdc290a5-d7cb-458d-a328-0e7aa97bb7df","Type":"ContainerStarted","Data":"434ecdbb8f60fa4a4472d95291a3485b2817e5f3da1a9889a4d72817a921564a"}
Jul 08 16:50:57.665662 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:57.665627    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:50:57.686424 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:57.686374    2565 generic.go:356] "Generic (PLEG): container finished" podID="457297bb-276a-4629-9b56-e4e6033e9d3b" containerID="fc8195358770e1cdb75b9b04dab181804d33821f6b6e4b3c67c26ada5e725145" exitCode=0
Jul 08 16:50:57.686816 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:57.686462    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc" event={"ID":"457297bb-276a-4629-9b56-e4e6033e9d3b","Type":"ContainerDied","Data":"fc8195358770e1cdb75b9b04dab181804d33821f6b6e4b3c67c26ada5e725145"}
Jul 08 16:50:58.694695 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:58.694637    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc" event={"ID":"457297bb-276a-4629-9b56-e4e6033e9d3b","Type":"ContainerStarted","Data":"9ed1e529179c61d82b4b604ffd3b49940af97f20e4404b3a36ecb659c3670350"}
Jul 08 16:50:58.695165 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:58.694717    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc"
Jul 08 16:50:58.718138 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:50:58.718078    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc" podStartSLOduration=2.942092231 podStartE2EDuration="3.718060652s" totalImagesPullingTime="0s" totalInitContainerRuntime="775.968421ms" isStatefulPod=false podCreationTimestamp="2026-07-08 16:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:50:58.715733211 +0000 UTC m=+995.429094637" watchObservedRunningTime="2026-07-08 16:50:58.718060652 +0000 UTC m=+995.431422102"
Jul 08 16:51:02.718865 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:02.718827    2565 generic.go:356] "Generic (PLEG): container finished" podID="cdc290a5-d7cb-458d-a328-0e7aa97bb7df" containerID="3c6b9906fe33b2b1794371d9c4dcc93fa5e961b017b2221b90b49d2ea16dbaa3" exitCode=0
Jul 08 16:51:02.719254 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:02.718896    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz" event={"ID":"cdc290a5-d7cb-458d-a328-0e7aa97bb7df","Type":"ContainerDied","Data":"3c6b9906fe33b2b1794371d9c4dcc93fa5e961b017b2221b90b49d2ea16dbaa3"}
Jul 08 16:51:03.728471 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:03.728434    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz" event={"ID":"cdc290a5-d7cb-458d-a328-0e7aa97bb7df","Type":"ContainerStarted","Data":"4369c0010853a026f5b6c8f304aab2e12197b75c2f194625c2f675365182af0b"}
Jul 08 16:51:03.766386 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:03.766304    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz" podStartSLOduration=2.24907375 podStartE2EDuration="8.766278795s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.517205045s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:51:03.762634895 +0000 UTC m=+1000.475996356" watchObservedRunningTime="2026-07-08 16:51:03.766278795 +0000 UTC m=+1000.479640222"
Jul 08 16:51:04.612249 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:04.612206    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"
Jul 08 16:51:05.611938 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:05.611899    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"
Jul 08 16:51:05.613894 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:05.613855    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz" podUID="cdc290a5-d7cb-458d-a328-0e7aa97bb7df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 08 16:51:11.416933 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:11.416895    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:51:14.608127 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:14.608080    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8"
Jul 08 16:51:15.612856 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:15.612770    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz" podUID="cdc290a5-d7cb-458d-a328-0e7aa97bb7df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 08 16:51:25.612879 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:25.612833    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz" podUID="cdc290a5-d7cb-458d-a328-0e7aa97bb7df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 08 16:51:28.280280 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:28.280241    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:51:29.701501 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:29.701473    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc"
Jul 08 16:51:33.369943 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:33.369905    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:51:33.398011 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:33.397958    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8"]
Jul 08 16:51:33.407502 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:51:33.407459    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 16:51:33.407502 ip-10-0-128-142 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 16:51:33.407502 ip-10-0-128-142 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 08 16:51:33.407502 ip-10-0-128-142 kubenswrapper[2565]:  > containerID="8bf99c6a2c2d4d41ac1b1e6dac2c4c13c07288e37e517a930325ef1f96b1a720" cmd=["/bin/sleep","15"]
Jul 08 16:51:33.407888 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:33.407513    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8" message="" err=<
Jul 08 16:51:33.407888 ip-10-0-128-142 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 16:51:33.407888 ip-10-0-128-142 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 08 16:51:33.407888 ip-10-0-128-142 kubenswrapper[2565]:  >
Jul 08 16:51:33.407888 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:51:33.407529    2565 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 16:51:33.407888 ip-10-0-128-142 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 16:51:33.407888 ip-10-0-128-142 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 08 16:51:33.407888 ip-10-0-128-142 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8" podUID="55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90" containerName="main" containerID="cri-o://8bf99c6a2c2d4d41ac1b1e6dac2c4c13c07288e37e517a930325ef1f96b1a720"
Jul 08 16:51:33.407888 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:33.407574    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8" podUID="55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90" containerName="main" containerID="cri-o://8bf99c6a2c2d4d41ac1b1e6dac2c4c13c07288e37e517a930325ef1f96b1a720" gracePeriod=60
Jul 08 16:51:33.525303 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:33.525262    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:51:33.667655 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:33.667613    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8"
Jul 08 16:51:33.826954 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:33.826926    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90-tls-certs\") pod \"55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90\" (UID: \"55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90\") "
Jul 08 16:51:33.827183 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:33.827038    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90-tokenizer-uds\") pod \"55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90\" (UID: \"55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90\") "
Jul 08 16:51:33.827183 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:33.827074    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bzdz\" (UniqueName: \"kubernetes.io/projected/55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90-kube-api-access-5bzdz\") pod \"55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90\" (UID: \"55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90\") "
Jul 08 16:51:33.827639 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:33.827596    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90" (UID: "55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:51:33.829287 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:33.829264    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90-kube-api-access-5bzdz" (OuterVolumeSpecName: "kube-api-access-5bzdz") pod "55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90" (UID: "55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90"). InnerVolumeSpecName "kube-api-access-5bzdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:51:33.829433 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:33.829415    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90" (UID: "55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:51:33.895111 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:33.895073    2565 generic.go:356] "Generic (PLEG): container finished" podID="55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90" containerID="8bf99c6a2c2d4d41ac1b1e6dac2c4c13c07288e37e517a930325ef1f96b1a720" exitCode=0
Jul 08 16:51:33.895295 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:33.895122    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8"
Jul 08 16:51:33.895295 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:33.895160    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8" event={"ID":"55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90","Type":"ContainerDied","Data":"8bf99c6a2c2d4d41ac1b1e6dac2c4c13c07288e37e517a930325ef1f96b1a720"}
Jul 08 16:51:33.895295 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:33.895216    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8" event={"ID":"55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90","Type":"ContainerDied","Data":"a9011d2e6e47e3da8ec9b0eb7a0c53a8b23e82be5782216251e86da0b68d19fa"}
Jul 08 16:51:33.895295 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:33.895245    2565 scope.go:117] "RemoveContainer" containerID="8bf99c6a2c2d4d41ac1b1e6dac2c4c13c07288e37e517a930325ef1f96b1a720"
Jul 08 16:51:33.905749 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:33.905731    2565 scope.go:117] "RemoveContainer" containerID="8bf99c6a2c2d4d41ac1b1e6dac2c4c13c07288e37e517a930325ef1f96b1a720"
Jul 08 16:51:33.906064 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:51:33.906047    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bf99c6a2c2d4d41ac1b1e6dac2c4c13c07288e37e517a930325ef1f96b1a720\": container with ID starting with 8bf99c6a2c2d4d41ac1b1e6dac2c4c13c07288e37e517a930325ef1f96b1a720 not found: ID does not exist" containerID="8bf99c6a2c2d4d41ac1b1e6dac2c4c13c07288e37e517a930325ef1f96b1a720"
Jul 08 16:51:33.906135 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:33.906070    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bf99c6a2c2d4d41ac1b1e6dac2c4c13c07288e37e517a930325ef1f96b1a720"} err="failed to get container status \"8bf99c6a2c2d4d41ac1b1e6dac2c4c13c07288e37e517a930325ef1f96b1a720\": rpc error: code = NotFound desc = could not find container \"8bf99c6a2c2d4d41ac1b1e6dac2c4c13c07288e37e517a930325ef1f96b1a720\": container with ID starting with 8bf99c6a2c2d4d41ac1b1e6dac2c4c13c07288e37e517a930325ef1f96b1a720 not found: ID does not exist"
Jul 08 16:51:33.920795 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:33.920758    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8"]
Jul 08 16:51:33.925521 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:33.925492    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvcb11a9c9832b99b016bc8f8e0ea095712-kserve-router-sche82tj8"]
Jul 08 16:51:33.928714 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:33.928661    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:51:33.928714 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:33.928712    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90-tokenizer-uds\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:51:33.928714 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:33.928722    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bzdz\" (UniqueName: \"kubernetes.io/projected/55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90-kube-api-access-5bzdz\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:51:35.612117 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:35.612069    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz" podUID="cdc290a5-d7cb-458d-a328-0e7aa97bb7df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 08 16:51:35.822585 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:35.822541    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90" path="/var/lib/kubelet/pods/55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90/volumes"
Jul 08 16:51:45.612149 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:45.612108    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz" podUID="cdc290a5-d7cb-458d-a328-0e7aa97bb7df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 08 16:51:52.373835 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:52.373795    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:51:52.441407 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:52.441366    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:51:55.612207 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:51:55.612149    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz" podUID="cdc290a5-d7cb-458d-a328-0e7aa97bb7df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 08 16:52:05.612547 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:52:05.612494    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz" podUID="cdc290a5-d7cb-458d-a328-0e7aa97bb7df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 08 16:52:15.612140 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:52:15.612041    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz" podUID="cdc290a5-d7cb-458d-a328-0e7aa97bb7df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 08 16:52:15.669481 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:52:15.669444    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:52:16.180138 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:52:16.180105    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:52:24.119031 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:52:24.118927    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:52:25.612658 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:52:25.612616    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz" podUID="cdc290a5-d7cb-458d-a328-0e7aa97bb7df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 08 16:52:35.612576 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:52:35.612531    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz" podUID="cdc290a5-d7cb-458d-a328-0e7aa97bb7df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 08 16:52:45.612256 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:52:45.612212    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz" podUID="cdc290a5-d7cb-458d-a328-0e7aa97bb7df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 08 16:52:55.612205 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:52:55.612163    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz" podUID="cdc290a5-d7cb-458d-a328-0e7aa97bb7df" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/health\": dial tcp 10.133.0.51:8000: connect: connection refused"
Jul 08 16:53:05.623083 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:05.623047    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"
Jul 08 16:53:05.631392 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:05.631364    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"
Jul 08 16:53:07.638349 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:07.638307    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"]
Jul 08 16:53:07.644599 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:07.644561    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc"]
Jul 08 16:53:07.654103 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:53:07.654060    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 16:53:07.654103 ip-10-0-128-142 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 16:53:07.654103 ip-10-0-128-142 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 08 16:53:07.654103 ip-10-0-128-142 kubenswrapper[2565]:  > containerID="9ed1e529179c61d82b4b604ffd3b49940af97f20e4404b3a36ecb659c3670350" cmd=["/bin/sleep","15"]
Jul 08 16:53:07.654381 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:07.654124    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc" message="" err=<
Jul 08 16:53:07.654381 ip-10-0-128-142 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 16:53:07.654381 ip-10-0-128-142 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 08 16:53:07.654381 ip-10-0-128-142 kubenswrapper[2565]:  >
Jul 08 16:53:07.654381 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:53:07.654142    2565 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 16:53:07.654381 ip-10-0-128-142 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 16:53:07.654381 ip-10-0-128-142 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 08 16:53:07.654381 ip-10-0-128-142 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc" podUID="457297bb-276a-4629-9b56-e4e6033e9d3b" containerName="main" containerID="cri-o://9ed1e529179c61d82b4b604ffd3b49940af97f20e4404b3a36ecb659c3670350"
Jul 08 16:53:07.654381 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:07.654190    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc" podUID="457297bb-276a-4629-9b56-e4e6033e9d3b" containerName="main" containerID="cri-o://9ed1e529179c61d82b4b604ffd3b49940af97f20e4404b3a36ecb659c3670350" gracePeriod=60
Jul 08 16:53:07.921985 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:07.921954    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc"
Jul 08 16:53:08.009592 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:08.009555    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/457297bb-276a-4629-9b56-e4e6033e9d3b-tokenizer-uds\") pod \"457297bb-276a-4629-9b56-e4e6033e9d3b\" (UID: \"457297bb-276a-4629-9b56-e4e6033e9d3b\") "
Jul 08 16:53:08.009592 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:08.009591    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/457297bb-276a-4629-9b56-e4e6033e9d3b-tls-certs\") pod \"457297bb-276a-4629-9b56-e4e6033e9d3b\" (UID: \"457297bb-276a-4629-9b56-e4e6033e9d3b\") "
Jul 08 16:53:08.009873 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:08.009642    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6zbr\" (UniqueName: \"kubernetes.io/projected/457297bb-276a-4629-9b56-e4e6033e9d3b-kube-api-access-g6zbr\") pod \"457297bb-276a-4629-9b56-e4e6033e9d3b\" (UID: \"457297bb-276a-4629-9b56-e4e6033e9d3b\") "
Jul 08 16:53:08.009873 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:08.009664    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/457297bb-276a-4629-9b56-e4e6033e9d3b-kserve-provision-location\") pod \"457297bb-276a-4629-9b56-e4e6033e9d3b\" (UID: \"457297bb-276a-4629-9b56-e4e6033e9d3b\") "
Jul 08 16:53:08.009993 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:08.009863    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/457297bb-276a-4629-9b56-e4e6033e9d3b-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "457297bb-276a-4629-9b56-e4e6033e9d3b" (UID: "457297bb-276a-4629-9b56-e4e6033e9d3b"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:53:08.010056 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:08.010002    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/457297bb-276a-4629-9b56-e4e6033e9d3b-tokenizer-uds\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:53:08.010493 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:08.010467    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/457297bb-276a-4629-9b56-e4e6033e9d3b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "457297bb-276a-4629-9b56-e4e6033e9d3b" (UID: "457297bb-276a-4629-9b56-e4e6033e9d3b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:53:08.011831 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:08.011810    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/457297bb-276a-4629-9b56-e4e6033e9d3b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "457297bb-276a-4629-9b56-e4e6033e9d3b" (UID: "457297bb-276a-4629-9b56-e4e6033e9d3b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:53:08.011929 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:08.011835    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457297bb-276a-4629-9b56-e4e6033e9d3b-kube-api-access-g6zbr" (OuterVolumeSpecName: "kube-api-access-g6zbr") pod "457297bb-276a-4629-9b56-e4e6033e9d3b" (UID: "457297bb-276a-4629-9b56-e4e6033e9d3b"). InnerVolumeSpecName "kube-api-access-g6zbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:53:08.041344 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:08.041284    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:53:08.110435 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:08.110397    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/457297bb-276a-4629-9b56-e4e6033e9d3b-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:53:08.110435 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:08.110429    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6zbr\" (UniqueName: \"kubernetes.io/projected/457297bb-276a-4629-9b56-e4e6033e9d3b-kube-api-access-g6zbr\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:53:08.110647 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:08.110445    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/457297bb-276a-4629-9b56-e4e6033e9d3b-kserve-provision-location\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:53:08.141777 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:08.141739    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:53:08.437340 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:08.437300    2565 generic.go:356] "Generic (PLEG): container finished" podID="457297bb-276a-4629-9b56-e4e6033e9d3b" containerID="9ed1e529179c61d82b4b604ffd3b49940af97f20e4404b3a36ecb659c3670350" exitCode=0
Jul 08 16:53:08.437531 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:08.437355    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc"
Jul 08 16:53:08.437531 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:08.437377    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc" event={"ID":"457297bb-276a-4629-9b56-e4e6033e9d3b","Type":"ContainerDied","Data":"9ed1e529179c61d82b4b604ffd3b49940af97f20e4404b3a36ecb659c3670350"}
Jul 08 16:53:08.437531 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:08.437440    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc" event={"ID":"457297bb-276a-4629-9b56-e4e6033e9d3b","Type":"ContainerDied","Data":"c1be136377b9bde399c2614d0efd4fb11e87a22f988adca88c6b8a8f9fb96719"}
Jul 08 16:53:08.437531 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:08.437471    2565 scope.go:117] "RemoveContainer" containerID="9ed1e529179c61d82b4b604ffd3b49940af97f20e4404b3a36ecb659c3670350"
Jul 08 16:53:08.447609 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:08.447588    2565 scope.go:117] "RemoveContainer" containerID="fc8195358770e1cdb75b9b04dab181804d33821f6b6e4b3c67c26ada5e725145"
Jul 08 16:53:08.457308 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:08.457285    2565 scope.go:117] "RemoveContainer" containerID="9ed1e529179c61d82b4b604ffd3b49940af97f20e4404b3a36ecb659c3670350"
Jul 08 16:53:08.457644 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:53:08.457617    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ed1e529179c61d82b4b604ffd3b49940af97f20e4404b3a36ecb659c3670350\": container with ID starting with 9ed1e529179c61d82b4b604ffd3b49940af97f20e4404b3a36ecb659c3670350 not found: ID does not exist" containerID="9ed1e529179c61d82b4b604ffd3b49940af97f20e4404b3a36ecb659c3670350"
Jul 08 16:53:08.457780 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:08.457656    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ed1e529179c61d82b4b604ffd3b49940af97f20e4404b3a36ecb659c3670350"} err="failed to get container status \"9ed1e529179c61d82b4b604ffd3b49940af97f20e4404b3a36ecb659c3670350\": rpc error: code = NotFound desc = could not find container \"9ed1e529179c61d82b4b604ffd3b49940af97f20e4404b3a36ecb659c3670350\": container with ID starting with 9ed1e529179c61d82b4b604ffd3b49940af97f20e4404b3a36ecb659c3670350 not found: ID does not exist"
Jul 08 16:53:08.457780 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:08.457700    2565 scope.go:117] "RemoveContainer" containerID="fc8195358770e1cdb75b9b04dab181804d33821f6b6e4b3c67c26ada5e725145"
Jul 08 16:53:08.457936 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:53:08.457919    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc8195358770e1cdb75b9b04dab181804d33821f6b6e4b3c67c26ada5e725145\": container with ID starting with fc8195358770e1cdb75b9b04dab181804d33821f6b6e4b3c67c26ada5e725145 not found: ID does not exist" containerID="fc8195358770e1cdb75b9b04dab181804d33821f6b6e4b3c67c26ada5e725145"
Jul 08 16:53:08.458033 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:08.457937    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc8195358770e1cdb75b9b04dab181804d33821f6b6e4b3c67c26ada5e725145"} err="failed to get container status \"fc8195358770e1cdb75b9b04dab181804d33821f6b6e4b3c67c26ada5e725145\": rpc error: code = NotFound desc = could not find container \"fc8195358770e1cdb75b9b04dab181804d33821f6b6e4b3c67c26ada5e725145\": container with ID starting with fc8195358770e1cdb75b9b04dab181804d33821f6b6e4b3c67c26ada5e725145 not found: ID does not exist"
Jul 08 16:53:08.464322 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:08.464295    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc"]
Jul 08 16:53:08.470760 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:08.470734    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-router-scheduler-5478bf674ccw6tc"]
Jul 08 16:53:08.925305 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:08.925268    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:53:09.822876 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:09.822842    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="457297bb-276a-4629-9b56-e4e6033e9d3b" path="/var/lib/kubelet/pods/457297bb-276a-4629-9b56-e4e6033e9d3b/volumes"
Jul 08 16:53:22.665189 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:22.665141    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz" podUID="cdc290a5-d7cb-458d-a328-0e7aa97bb7df" containerName="main" containerID="cri-o://4369c0010853a026f5b6c8f304aab2e12197b75c2f194625c2f675365182af0b" gracePeriod=45
Jul 08 16:53:23.421506 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.421477    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"
Jul 08 16:53:23.523810 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.523707    2565 generic.go:356] "Generic (PLEG): container finished" podID="cdc290a5-d7cb-458d-a328-0e7aa97bb7df" containerID="4369c0010853a026f5b6c8f304aab2e12197b75c2f194625c2f675365182af0b" exitCode=0
Jul 08 16:53:23.523810 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.523736    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz" event={"ID":"cdc290a5-d7cb-458d-a328-0e7aa97bb7df","Type":"ContainerDied","Data":"4369c0010853a026f5b6c8f304aab2e12197b75c2f194625c2f675365182af0b"}
Jul 08 16:53:23.523810 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.523756    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"
Jul 08 16:53:23.523810 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.523809    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz" event={"ID":"cdc290a5-d7cb-458d-a328-0e7aa97bb7df","Type":"ContainerDied","Data":"434ecdbb8f60fa4a4472d95291a3485b2817e5f3da1a9889a4d72817a921564a"}
Jul 08 16:53:23.524161 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.523841    2565 scope.go:117] "RemoveContainer" containerID="4369c0010853a026f5b6c8f304aab2e12197b75c2f194625c2f675365182af0b"
Jul 08 16:53:23.533224 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.533198    2565 scope.go:117] "RemoveContainer" containerID="3c6b9906fe33b2b1794371d9c4dcc93fa5e961b017b2221b90b49d2ea16dbaa3"
Jul 08 16:53:23.542347 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.542314    2565 scope.go:117] "RemoveContainer" containerID="4369c0010853a026f5b6c8f304aab2e12197b75c2f194625c2f675365182af0b"
Jul 08 16:53:23.542661 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:53:23.542642    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4369c0010853a026f5b6c8f304aab2e12197b75c2f194625c2f675365182af0b\": container with ID starting with 4369c0010853a026f5b6c8f304aab2e12197b75c2f194625c2f675365182af0b not found: ID does not exist" containerID="4369c0010853a026f5b6c8f304aab2e12197b75c2f194625c2f675365182af0b"
Jul 08 16:53:23.542738 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.542693    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4369c0010853a026f5b6c8f304aab2e12197b75c2f194625c2f675365182af0b"} err="failed to get container status \"4369c0010853a026f5b6c8f304aab2e12197b75c2f194625c2f675365182af0b\": rpc error: code = NotFound desc = could not find container \"4369c0010853a026f5b6c8f304aab2e12197b75c2f194625c2f675365182af0b\": container with ID starting with 4369c0010853a026f5b6c8f304aab2e12197b75c2f194625c2f675365182af0b not found: ID does not exist"
Jul 08 16:53:23.542738 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.542715    2565 scope.go:117] "RemoveContainer" containerID="3c6b9906fe33b2b1794371d9c4dcc93fa5e961b017b2221b90b49d2ea16dbaa3"
Jul 08 16:53:23.542970 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:53:23.542953    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c6b9906fe33b2b1794371d9c4dcc93fa5e961b017b2221b90b49d2ea16dbaa3\": container with ID starting with 3c6b9906fe33b2b1794371d9c4dcc93fa5e961b017b2221b90b49d2ea16dbaa3 not found: ID does not exist" containerID="3c6b9906fe33b2b1794371d9c4dcc93fa5e961b017b2221b90b49d2ea16dbaa3"
Jul 08 16:53:23.543023 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.542978    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c6b9906fe33b2b1794371d9c4dcc93fa5e961b017b2221b90b49d2ea16dbaa3"} err="failed to get container status \"3c6b9906fe33b2b1794371d9c4dcc93fa5e961b017b2221b90b49d2ea16dbaa3\": rpc error: code = NotFound desc = could not find container \"3c6b9906fe33b2b1794371d9c4dcc93fa5e961b017b2221b90b49d2ea16dbaa3\": container with ID starting with 3c6b9906fe33b2b1794371d9c4dcc93fa5e961b017b2221b90b49d2ea16dbaa3 not found: ID does not exist"
Jul 08 16:53:23.548340 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.548302    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-tmp-dir\") pod \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\" (UID: \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\") "
Jul 08 16:53:23.548453 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.548360    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-home\") pod \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\" (UID: \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\") "
Jul 08 16:53:23.548453 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.548386    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2mkp6\" (UniqueName: \"kubernetes.io/projected/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-kube-api-access-2mkp6\") pod \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\" (UID: \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\") "
Jul 08 16:53:23.548453 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.548436    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-tls-certs\") pod \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\" (UID: \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\") "
Jul 08 16:53:23.548631 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.548489    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-model-cache\") pod \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\" (UID: \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\") "
Jul 08 16:53:23.548631 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.548529    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-kserve-provision-location\") pod \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\" (UID: \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\") "
Jul 08 16:53:23.548631 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.548564    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-dshm\") pod \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\" (UID: \"cdc290a5-d7cb-458d-a328-0e7aa97bb7df\") "
Jul 08 16:53:23.548858 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.548830    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-model-cache" (OuterVolumeSpecName: "model-cache") pod "cdc290a5-d7cb-458d-a328-0e7aa97bb7df" (UID: "cdc290a5-d7cb-458d-a328-0e7aa97bb7df"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:53:23.549176 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.549139    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-home" (OuterVolumeSpecName: "home") pod "cdc290a5-d7cb-458d-a328-0e7aa97bb7df" (UID: "cdc290a5-d7cb-458d-a328-0e7aa97bb7df"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:53:23.550785 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.550746    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cdc290a5-d7cb-458d-a328-0e7aa97bb7df" (UID: "cdc290a5-d7cb-458d-a328-0e7aa97bb7df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:53:23.551117 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.551095    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-kube-api-access-2mkp6" (OuterVolumeSpecName: "kube-api-access-2mkp6") pod "cdc290a5-d7cb-458d-a328-0e7aa97bb7df" (UID: "cdc290a5-d7cb-458d-a328-0e7aa97bb7df"). InnerVolumeSpecName "kube-api-access-2mkp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:53:23.551249 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.551234    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-dshm" (OuterVolumeSpecName: "dshm") pod "cdc290a5-d7cb-458d-a328-0e7aa97bb7df" (UID: "cdc290a5-d7cb-458d-a328-0e7aa97bb7df"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:53:23.565916 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.565873    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cdc290a5-d7cb-458d-a328-0e7aa97bb7df" (UID: "cdc290a5-d7cb-458d-a328-0e7aa97bb7df"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:53:23.633923 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.633874    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cdc290a5-d7cb-458d-a328-0e7aa97bb7df" (UID: "cdc290a5-d7cb-458d-a328-0e7aa97bb7df"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:53:23.649404 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.649365    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:53:23.649404 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.649398    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-kserve-provision-location\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:53:23.649404 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.649409    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:53:23.649645 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.649440    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:53:23.649645 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.649451    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:53:23.649645 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.649459    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mkp6\" (UniqueName: \"kubernetes.io/projected/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-kube-api-access-2mkp6\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:53:23.649645 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.649467    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cdc290a5-d7cb-458d-a328-0e7aa97bb7df-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:53:23.854846 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.854815    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"]
Jul 08 16:53:23.860904 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:23.860853    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-invalid-token-test-kserve-69f89b8bf7-whpjz"]
Jul 08 16:53:25.822837 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:25.822792    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cdc290a5-d7cb-458d-a328-0e7aa97bb7df" path="/var/lib/kubelet/pods/cdc290a5-d7cb-458d-a328-0e7aa97bb7df/volumes"
Jul 08 16:53:30.527278 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.527238    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"]
Jul 08 16:53:30.527697 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.527660    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="457297bb-276a-4629-9b56-e4e6033e9d3b" containerName="main"
Jul 08 16:53:30.527697 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.527690    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="457297bb-276a-4629-9b56-e4e6033e9d3b" containerName="main"
Jul 08 16:53:30.527786 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.527701    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="457297bb-276a-4629-9b56-e4e6033e9d3b" containerName="storage-initializer"
Jul 08 16:53:30.527786 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.527709    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="457297bb-276a-4629-9b56-e4e6033e9d3b" containerName="storage-initializer"
Jul 08 16:53:30.527786 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.527716    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdc290a5-d7cb-458d-a328-0e7aa97bb7df" containerName="main"
Jul 08 16:53:30.527786 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.527722    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdc290a5-d7cb-458d-a328-0e7aa97bb7df" containerName="main"
Jul 08 16:53:30.527786 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.527733    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cdc290a5-d7cb-458d-a328-0e7aa97bb7df" containerName="storage-initializer"
Jul 08 16:53:30.527786 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.527739    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cdc290a5-d7cb-458d-a328-0e7aa97bb7df" containerName="storage-initializer"
Jul 08 16:53:30.527786 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.527757    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90" containerName="main"
Jul 08 16:53:30.527786 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.527762    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90" containerName="main"
Jul 08 16:53:30.528038 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.527821    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cdc290a5-d7cb-458d-a328-0e7aa97bb7df" containerName="main"
Jul 08 16:53:30.528038 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.527829    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="55e76bc0-af1b-421f-a4a6-f6dcb3e2ba90" containerName="main"
Jul 08 16:53:30.528038 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.527837    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="457297bb-276a-4629-9b56-e4e6033e9d3b" containerName="main"
Jul 08 16:53:30.531910 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.531887    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"
Jul 08 16:53:30.536633 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.536607    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qxdr\""
Jul 08 16:53:30.541410 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.541384    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-kserve-self-signed-certs\""
Jul 08 16:53:30.542028 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.542008    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 16:53:30.563311 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.563274    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"]
Jul 08 16:53:30.586934 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.586894    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:53:30.612355 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.612318    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-dshm\") pod \"auth-disabled-test-kserve-84ff994f99-sl6v4\" (UID: \"97052463-d959-4ece-8c21-5d267ccdebf2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"
Jul 08 16:53:30.612551 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.612381    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-tmp-dir\") pod \"auth-disabled-test-kserve-84ff994f99-sl6v4\" (UID: \"97052463-d959-4ece-8c21-5d267ccdebf2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"
Jul 08 16:53:30.612551 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.612423    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-home\") pod \"auth-disabled-test-kserve-84ff994f99-sl6v4\" (UID: \"97052463-d959-4ece-8c21-5d267ccdebf2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"
Jul 08 16:53:30.612551 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.612481    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97052463-d959-4ece-8c21-5d267ccdebf2-tls-certs\") pod \"auth-disabled-test-kserve-84ff994f99-sl6v4\" (UID: \"97052463-d959-4ece-8c21-5d267ccdebf2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"
Jul 08 16:53:30.612551 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.612527    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-model-cache\") pod \"auth-disabled-test-kserve-84ff994f99-sl6v4\" (UID: \"97052463-d959-4ece-8c21-5d267ccdebf2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"
Jul 08 16:53:30.612730 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.612553    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph9nj\" (UniqueName: \"kubernetes.io/projected/97052463-d959-4ece-8c21-5d267ccdebf2-kube-api-access-ph9nj\") pod \"auth-disabled-test-kserve-84ff994f99-sl6v4\" (UID: \"97052463-d959-4ece-8c21-5d267ccdebf2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"
Jul 08 16:53:30.612730 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.612615    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-kserve-provision-location\") pod \"auth-disabled-test-kserve-84ff994f99-sl6v4\" (UID: \"97052463-d959-4ece-8c21-5d267ccdebf2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"
Jul 08 16:53:30.713967 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.713932    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-dshm\") pod \"auth-disabled-test-kserve-84ff994f99-sl6v4\" (UID: \"97052463-d959-4ece-8c21-5d267ccdebf2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"
Jul 08 16:53:30.714173 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.713977    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-tmp-dir\") pod \"auth-disabled-test-kserve-84ff994f99-sl6v4\" (UID: \"97052463-d959-4ece-8c21-5d267ccdebf2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"
Jul 08 16:53:30.714173 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.714004    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-home\") pod \"auth-disabled-test-kserve-84ff994f99-sl6v4\" (UID: \"97052463-d959-4ece-8c21-5d267ccdebf2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"
Jul 08 16:53:30.714173 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.714023    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97052463-d959-4ece-8c21-5d267ccdebf2-tls-certs\") pod \"auth-disabled-test-kserve-84ff994f99-sl6v4\" (UID: \"97052463-d959-4ece-8c21-5d267ccdebf2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"
Jul 08 16:53:30.714173 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.714061    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-model-cache\") pod \"auth-disabled-test-kserve-84ff994f99-sl6v4\" (UID: \"97052463-d959-4ece-8c21-5d267ccdebf2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"
Jul 08 16:53:30.714173 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.714085    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph9nj\" (UniqueName: \"kubernetes.io/projected/97052463-d959-4ece-8c21-5d267ccdebf2-kube-api-access-ph9nj\") pod \"auth-disabled-test-kserve-84ff994f99-sl6v4\" (UID: \"97052463-d959-4ece-8c21-5d267ccdebf2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"
Jul 08 16:53:30.714173 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.714105    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-kserve-provision-location\") pod \"auth-disabled-test-kserve-84ff994f99-sl6v4\" (UID: \"97052463-d959-4ece-8c21-5d267ccdebf2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"
Jul 08 16:53:30.714522 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.714432    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-tmp-dir\") pod \"auth-disabled-test-kserve-84ff994f99-sl6v4\" (UID: \"97052463-d959-4ece-8c21-5d267ccdebf2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"
Jul 08 16:53:30.714522 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.714477    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-home\") pod \"auth-disabled-test-kserve-84ff994f99-sl6v4\" (UID: \"97052463-d959-4ece-8c21-5d267ccdebf2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"
Jul 08 16:53:30.714624 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.714530    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-model-cache\") pod \"auth-disabled-test-kserve-84ff994f99-sl6v4\" (UID: \"97052463-d959-4ece-8c21-5d267ccdebf2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"
Jul 08 16:53:30.714624 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.714582    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-kserve-provision-location\") pod \"auth-disabled-test-kserve-84ff994f99-sl6v4\" (UID: \"97052463-d959-4ece-8c21-5d267ccdebf2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"
Jul 08 16:53:30.716431 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.716406    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-dshm\") pod \"auth-disabled-test-kserve-84ff994f99-sl6v4\" (UID: \"97052463-d959-4ece-8c21-5d267ccdebf2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"
Jul 08 16:53:30.716652 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.716633    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97052463-d959-4ece-8c21-5d267ccdebf2-tls-certs\") pod \"auth-disabled-test-kserve-84ff994f99-sl6v4\" (UID: \"97052463-d959-4ece-8c21-5d267ccdebf2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"
Jul 08 16:53:30.727560 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.727530    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph9nj\" (UniqueName: \"kubernetes.io/projected/97052463-d959-4ece-8c21-5d267ccdebf2-kube-api-access-ph9nj\") pod \"auth-disabled-test-kserve-84ff994f99-sl6v4\" (UID: \"97052463-d959-4ece-8c21-5d267ccdebf2\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"
Jul 08 16:53:30.843080 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:30.842985    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"
Jul 08 16:53:31.011273 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:31.011244    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"]
Jul 08 16:53:31.013414 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:53:31.013386    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97052463_d959_4ece_8c21_5d267ccdebf2.slice/crio-2709d6ebad52bc7ad6729daf4a08f789f6fbaec2a04ef1c0b8111dbc4ded0aa7 WatchSource:0}: Error finding container 2709d6ebad52bc7ad6729daf4a08f789f6fbaec2a04ef1c0b8111dbc4ded0aa7: Status 404 returned error can't find the container with id 2709d6ebad52bc7ad6729daf4a08f789f6fbaec2a04ef1c0b8111dbc4ded0aa7
Jul 08 16:53:31.075366 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:31.075328    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz"]
Jul 08 16:53:31.080812 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:31.080787    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz"
Jul 08 16:53:31.089058 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:31.089033    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"auth-disabled-test-epp-sa-dockercfg-wcqg8\""
Jul 08 16:53:31.111060 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:31.110968    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz"]
Jul 08 16:53:31.144478 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:31.144440    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:53:31.218871 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:31.218832    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59595c67-9459-4614-8e29-836f7139515b-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz\" (UID: \"59595c67-9459-4614-8e29-836f7139515b\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz"
Jul 08 16:53:31.219051 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:31.218889    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jl9f\" (UniqueName: \"kubernetes.io/projected/59595c67-9459-4614-8e29-836f7139515b-kube-api-access-2jl9f\") pod \"auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz\" (UID: \"59595c67-9459-4614-8e29-836f7139515b\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz"
Jul 08 16:53:31.219051 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:31.218959    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/59595c67-9459-4614-8e29-836f7139515b-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz\" (UID: \"59595c67-9459-4614-8e29-836f7139515b\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz"
Jul 08 16:53:31.219051 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:31.219023    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59595c67-9459-4614-8e29-836f7139515b-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz\" (UID: \"59595c67-9459-4614-8e29-836f7139515b\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz"
Jul 08 16:53:31.319808 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:31.319769    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59595c67-9459-4614-8e29-836f7139515b-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz\" (UID: \"59595c67-9459-4614-8e29-836f7139515b\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz"
Jul 08 16:53:31.319996 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:31.319823    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jl9f\" (UniqueName: \"kubernetes.io/projected/59595c67-9459-4614-8e29-836f7139515b-kube-api-access-2jl9f\") pod \"auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz\" (UID: \"59595c67-9459-4614-8e29-836f7139515b\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz"
Jul 08 16:53:31.319996 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:31.319843    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/59595c67-9459-4614-8e29-836f7139515b-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz\" (UID: \"59595c67-9459-4614-8e29-836f7139515b\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz"
Jul 08 16:53:31.320132 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:31.320024    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59595c67-9459-4614-8e29-836f7139515b-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz\" (UID: \"59595c67-9459-4614-8e29-836f7139515b\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz"
Jul 08 16:53:31.320297 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:31.320274    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59595c67-9459-4614-8e29-836f7139515b-kserve-provision-location\") pod \"auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz\" (UID: \"59595c67-9459-4614-8e29-836f7139515b\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz"
Jul 08 16:53:31.320297 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:31.320289    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/59595c67-9459-4614-8e29-836f7139515b-tokenizer-uds\") pod \"auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz\" (UID: \"59595c67-9459-4614-8e29-836f7139515b\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz"
Jul 08 16:53:31.322446 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:31.322413    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59595c67-9459-4614-8e29-836f7139515b-tls-certs\") pod \"auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz\" (UID: \"59595c67-9459-4614-8e29-836f7139515b\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz"
Jul 08 16:53:31.331713 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:31.331667    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jl9f\" (UniqueName: \"kubernetes.io/projected/59595c67-9459-4614-8e29-836f7139515b-kube-api-access-2jl9f\") pod \"auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz\" (UID: \"59595c67-9459-4614-8e29-836f7139515b\") " pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz"
Jul 08 16:53:31.391478 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:31.391352    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz"
Jul 08 16:53:31.559983 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:31.559946    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz"]
Jul 08 16:53:31.560546 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:53:31.560525    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59595c67_9459_4614_8e29_836f7139515b.slice/crio-fd0d163e0ecebffcd300b42cd3997ba96d0b06520ed99ea89d1d08b9c7378994 WatchSource:0}: Error finding container fd0d163e0ecebffcd300b42cd3997ba96d0b06520ed99ea89d1d08b9c7378994: Status 404 returned error can't find the container with id fd0d163e0ecebffcd300b42cd3997ba96d0b06520ed99ea89d1d08b9c7378994
Jul 08 16:53:31.569831 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:31.569801    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz" event={"ID":"59595c67-9459-4614-8e29-836f7139515b","Type":"ContainerStarted","Data":"fd0d163e0ecebffcd300b42cd3997ba96d0b06520ed99ea89d1d08b9c7378994"}
Jul 08 16:53:31.571604 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:31.571577    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4" event={"ID":"97052463-d959-4ece-8c21-5d267ccdebf2","Type":"ContainerStarted","Data":"3259a4d7dc041e28be39d76e5d377f287f1183e0453f1af416ff867c13c87f42"}
Jul 08 16:53:31.571754 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:31.571640    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4" event={"ID":"97052463-d959-4ece-8c21-5d267ccdebf2","Type":"ContainerStarted","Data":"2709d6ebad52bc7ad6729daf4a08f789f6fbaec2a04ef1c0b8111dbc4ded0aa7"}
Jul 08 16:53:32.580976 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:32.580938    2565 generic.go:356] "Generic (PLEG): container finished" podID="59595c67-9459-4614-8e29-836f7139515b" containerID="e3bcf3b4e4794c8ad03bef48362c04834c103d7042d99c0781a48f4c1533cfc1" exitCode=0
Jul 08 16:53:32.581482 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:32.581032    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz" event={"ID":"59595c67-9459-4614-8e29-836f7139515b","Type":"ContainerDied","Data":"e3bcf3b4e4794c8ad03bef48362c04834c103d7042d99c0781a48f4c1533cfc1"}
Jul 08 16:53:33.596998 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:33.596873    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz" event={"ID":"59595c67-9459-4614-8e29-836f7139515b","Type":"ContainerStarted","Data":"5eb9b8e091aacf458d76fc956c10d7d3d453d7a64f338e5d464d212f737489dc"}
Jul 08 16:53:33.597961 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:33.597071    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz"
Jul 08 16:53:33.623661 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:33.623596    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz" podStartSLOduration=1.7822111139999999 podStartE2EDuration="2.623573064s" totalImagesPullingTime="0s" totalInitContainerRuntime="841.36195ms" isStatefulPod=false podCreationTimestamp="2026-07-08 16:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:53:33.623204487 +0000 UTC m=+1150.336565914" watchObservedRunningTime="2026-07-08 16:53:33.623573064 +0000 UTC m=+1150.336934490"
Jul 08 16:53:36.623394 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:36.623297    2565 generic.go:356] "Generic (PLEG): container finished" podID="97052463-d959-4ece-8c21-5d267ccdebf2" containerID="3259a4d7dc041e28be39d76e5d377f287f1183e0453f1af416ff867c13c87f42" exitCode=0
Jul 08 16:53:36.623394 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:36.623367    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4" event={"ID":"97052463-d959-4ece-8c21-5d267ccdebf2","Type":"ContainerDied","Data":"3259a4d7dc041e28be39d76e5d377f287f1183e0453f1af416ff867c13c87f42"}
Jul 08 16:53:37.632071 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:37.632030    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4" event={"ID":"97052463-d959-4ece-8c21-5d267ccdebf2","Type":"ContainerStarted","Data":"3cd7f8ca6a43ee712d823c1f294f00b909b97b7feddb1f3e4c7ec306251e8285"}
Jul 08 16:53:37.659656 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:37.659581    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4" podStartSLOduration=2.512495372 podStartE2EDuration="7.659559823s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.147064451s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:53:37.658292768 +0000 UTC m=+1154.371654196" watchObservedRunningTime="2026-07-08 16:53:37.659559823 +0000 UTC m=+1154.372921250"
Jul 08 16:53:37.699308 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:37.699259    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:53:37.843547 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:37.843510    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"
Jul 08 16:53:40.843577 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:40.843541    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"
Jul 08 16:53:40.845249 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:40.845219    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4" podUID="97052463-d959-4ece-8c21-5d267ccdebf2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 08 16:53:50.844190 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:53:50.844091    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4" podUID="97052463-d959-4ece-8c21-5d267ccdebf2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 08 16:54:00.843763 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:54:00.843703    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4" podUID="97052463-d959-4ece-8c21-5d267ccdebf2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 08 16:54:02.093218 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:54:02.093178    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:54:02.454249 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:54:02.454214    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:54:04.606127 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:54:04.606095    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz"
Jul 08 16:54:10.843938 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:54:10.843893    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4" podUID="97052463-d959-4ece-8c21-5d267ccdebf2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 08 16:54:20.844008 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:54:20.843953    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4" podUID="97052463-d959-4ece-8c21-5d267ccdebf2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 08 16:54:30.843487 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:54:30.843432    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4" podUID="97052463-d959-4ece-8c21-5d267ccdebf2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 08 16:54:30.921046 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:54:30.921005    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:54:37.529761 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:54:37.529717    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:54:37.634744 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:54:37.634706    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:54:40.843990 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:54:40.843946    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4" podUID="97052463-d959-4ece-8c21-5d267ccdebf2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 08 16:54:50.844007 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:54:50.843956    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4" podUID="97052463-d959-4ece-8c21-5d267ccdebf2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 08 16:54:57.317693 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:54:57.317634    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:54:57.402151 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:54:57.402111    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:55:00.844019 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:00.843969    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4" podUID="97052463-d959-4ece-8c21-5d267ccdebf2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 08 16:55:10.843586 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:10.843535    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4" podUID="97052463-d959-4ece-8c21-5d267ccdebf2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 08 16:55:20.843955 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:20.843904    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4" podUID="97052463-d959-4ece-8c21-5d267ccdebf2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 08 16:55:27.301475 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.301432    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"]
Jul 08 16:55:27.306081 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.306053    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:55:27.308932 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.308908    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv3e414c2ba058a022dfd694dbcbac5b51-kserve-self-signed-certs\""
Jul 08 16:55:27.308932 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.308919    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 08 16:55:27.318840 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.318809    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"]
Jul 08 16:55:27.380963 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.380924    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:55:27.468333 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.468295    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d5691ff-d2c2-4559-82de-70d5dccdf797-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:55:27.468543 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.468388    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d5691ff-d2c2-4559-82de-70d5dccdf797-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:55:27.468543 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.468464    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:55:27.468639 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.468531    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:55:27.468639 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.468575    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:55:27.468639 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.468605    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwpn9\" (UniqueName: \"kubernetes.io/projected/5d5691ff-d2c2-4559-82de-70d5dccdf797-kube-api-access-hwpn9\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:55:27.468792 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.468655    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:55:27.468792 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.468701    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:55:27.570102 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.570005    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d5691ff-d2c2-4559-82de-70d5dccdf797-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:55:27.570102 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.570073    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:55:27.570102 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.570099    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:55:27.570389 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.570118    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:55:27.570389 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.570276    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwpn9\" (UniqueName: \"kubernetes.io/projected/5d5691ff-d2c2-4559-82de-70d5dccdf797-kube-api-access-hwpn9\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:55:27.570389 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.570347    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:55:27.570389 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.570382    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:55:27.570668 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.570450    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d5691ff-d2c2-4559-82de-70d5dccdf797-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:55:27.570668 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.570512    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:55:27.570668 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.570565    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-home\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:55:27.570668 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.570624    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:55:27.570850 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.570839    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:55:27.571237 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.571208    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d5691ff-d2c2-4559-82de-70d5dccdf797-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:55:27.572743 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.572713    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-dshm\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:55:27.572930 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.572910    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d5691ff-d2c2-4559-82de-70d5dccdf797-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:55:27.581181 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.581142    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwpn9\" (UniqueName: \"kubernetes.io/projected/5d5691ff-d2c2-4559-82de-70d5dccdf797-kube-api-access-hwpn9\") pod \"llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:55:27.617995 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.617954    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:55:27.695285 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.695150    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:55:27.786024 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:27.785981    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"]
Jul 08 16:55:27.786570 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:55:27.786543    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d5691ff_d2c2_4559_82de_70d5dccdf797.slice/crio-855cb77f24fc9a34cac901073e2cbc9b3c34a0a269d1b6d6793af5f0309d453b WatchSource:0}: Error finding container 855cb77f24fc9a34cac901073e2cbc9b3c34a0a269d1b6d6793af5f0309d453b: Status 404 returned error can't find the container with id 855cb77f24fc9a34cac901073e2cbc9b3c34a0a269d1b6d6793af5f0309d453b
Jul 08 16:55:28.297202 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:28.297159    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986" event={"ID":"5d5691ff-d2c2-4559-82de-70d5dccdf797","Type":"ContainerStarted","Data":"831c3cb341971c18ff935c13122ef81216044f3f38993abc36ff00f44fafaae0"}
Jul 08 16:55:28.297399 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:28.297256    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986" event={"ID":"5d5691ff-d2c2-4559-82de-70d5dccdf797","Type":"ContainerStarted","Data":"855cb77f24fc9a34cac901073e2cbc9b3c34a0a269d1b6d6793af5f0309d453b"}
Jul 08 16:55:30.844541 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:30.844490    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4" podUID="97052463-d959-4ece-8c21-5d267ccdebf2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 08 16:55:33.332170 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:33.332065    2565 generic.go:356] "Generic (PLEG): container finished" podID="5d5691ff-d2c2-4559-82de-70d5dccdf797" containerID="831c3cb341971c18ff935c13122ef81216044f3f38993abc36ff00f44fafaae0" exitCode=0
Jul 08 16:55:33.332170 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:33.332149    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986" event={"ID":"5d5691ff-d2c2-4559-82de-70d5dccdf797","Type":"ContainerDied","Data":"831c3cb341971c18ff935c13122ef81216044f3f38993abc36ff00f44fafaae0"}
Jul 08 16:55:34.342298 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:34.342265    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986" event={"ID":"5d5691ff-d2c2-4559-82de-70d5dccdf797","Type":"ContainerStarted","Data":"d02da8a4f791f4edae66cf9244a39610a37542252d1ac3b8e409f856ab3888b5"}
Jul 08 16:55:34.402843 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:34.402785    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986" podStartSLOduration=2.333264159 podStartE2EDuration="7.402764902s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.069500743s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:55:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:55:34.399314826 +0000 UTC m=+1271.112676255" watchObservedRunningTime="2026-07-08 16:55:34.402764902 +0000 UTC m=+1271.116126346"
Jul 08 16:55:34.619189 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:34.619092    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:55:36.827553 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:36.827514    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:55:37.618835 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:37.618798    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:55:37.620738 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:37.620712    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986" podUID="5d5691ff-d2c2-4559-82de-70d5dccdf797" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 08 16:55:40.853998 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:40.853967    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"
Jul 08 16:55:40.863372 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:40.863339    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"
Jul 08 16:55:46.432926 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:46.432885    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz"]
Jul 08 16:55:46.436491 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:46.436453    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"]
Jul 08 16:55:46.442175 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:46.442142    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:55:46.444224 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:55:46.444188    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 16:55:46.444224 ip-10-0-128-142 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 16:55:46.444224 ip-10-0-128-142 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 08 16:55:46.444224 ip-10-0-128-142 kubenswrapper[2565]:  > containerID="5eb9b8e091aacf458d76fc956c10d7d3d453d7a64f338e5d464d212f737489dc" cmd=["/bin/sleep","15"]
Jul 08 16:55:46.444521 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:46.444240    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz" message="" err=<
Jul 08 16:55:46.444521 ip-10-0-128-142 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 16:55:46.444521 ip-10-0-128-142 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 08 16:55:46.444521 ip-10-0-128-142 kubenswrapper[2565]:  >
Jul 08 16:55:46.444521 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:55:46.444260    2565 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 16:55:46.444521 ip-10-0-128-142 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 16:55:46.444521 ip-10-0-128-142 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 08 16:55:46.444521 ip-10-0-128-142 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz" podUID="59595c67-9459-4614-8e29-836f7139515b" containerName="main" containerID="cri-o://5eb9b8e091aacf458d76fc956c10d7d3d453d7a64f338e5d464d212f737489dc"
Jul 08 16:55:46.444521 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:46.444321    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz" podUID="59595c67-9459-4614-8e29-836f7139515b" containerName="main" containerID="cri-o://5eb9b8e091aacf458d76fc956c10d7d3d453d7a64f338e5d464d212f737489dc" gracePeriod=60
Jul 08 16:55:46.525270 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:46.525232    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:55:46.705849 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:46.705825    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz"
Jul 08 16:55:46.766694 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:46.766652    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59595c67-9459-4614-8e29-836f7139515b-kserve-provision-location\") pod \"59595c67-9459-4614-8e29-836f7139515b\" (UID: \"59595c67-9459-4614-8e29-836f7139515b\") "
Jul 08 16:55:46.766870 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:46.766790    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59595c67-9459-4614-8e29-836f7139515b-tls-certs\") pod \"59595c67-9459-4614-8e29-836f7139515b\" (UID: \"59595c67-9459-4614-8e29-836f7139515b\") "
Jul 08 16:55:46.766870 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:46.766831    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jl9f\" (UniqueName: \"kubernetes.io/projected/59595c67-9459-4614-8e29-836f7139515b-kube-api-access-2jl9f\") pod \"59595c67-9459-4614-8e29-836f7139515b\" (UID: \"59595c67-9459-4614-8e29-836f7139515b\") "
Jul 08 16:55:46.766995 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:46.766870    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/59595c67-9459-4614-8e29-836f7139515b-tokenizer-uds\") pod \"59595c67-9459-4614-8e29-836f7139515b\" (UID: \"59595c67-9459-4614-8e29-836f7139515b\") "
Jul 08 16:55:46.767510 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:46.767484    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59595c67-9459-4614-8e29-836f7139515b-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "59595c67-9459-4614-8e29-836f7139515b" (UID: "59595c67-9459-4614-8e29-836f7139515b"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:55:46.767700 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:46.767512    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59595c67-9459-4614-8e29-836f7139515b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "59595c67-9459-4614-8e29-836f7139515b" (UID: "59595c67-9459-4614-8e29-836f7139515b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:55:46.769481 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:46.769453    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59595c67-9459-4614-8e29-836f7139515b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "59595c67-9459-4614-8e29-836f7139515b" (UID: "59595c67-9459-4614-8e29-836f7139515b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:55:46.769481 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:46.769452    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59595c67-9459-4614-8e29-836f7139515b-kube-api-access-2jl9f" (OuterVolumeSpecName: "kube-api-access-2jl9f") pod "59595c67-9459-4614-8e29-836f7139515b" (UID: "59595c67-9459-4614-8e29-836f7139515b"). InnerVolumeSpecName "kube-api-access-2jl9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:55:46.868376 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:46.868327    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/59595c67-9459-4614-8e29-836f7139515b-kserve-provision-location\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:55:46.868376 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:46.868373    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59595c67-9459-4614-8e29-836f7139515b-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:55:46.868600 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:46.868389    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jl9f\" (UniqueName: \"kubernetes.io/projected/59595c67-9459-4614-8e29-836f7139515b-kube-api-access-2jl9f\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:55:46.868600 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:46.868403    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/59595c67-9459-4614-8e29-836f7139515b-tokenizer-uds\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:55:47.427128 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:47.427091    2565 generic.go:356] "Generic (PLEG): container finished" podID="59595c67-9459-4614-8e29-836f7139515b" containerID="5eb9b8e091aacf458d76fc956c10d7d3d453d7a64f338e5d464d212f737489dc" exitCode=0
Jul 08 16:55:47.427338 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:47.427154    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz"
Jul 08 16:55:47.427338 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:47.427180    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz" event={"ID":"59595c67-9459-4614-8e29-836f7139515b","Type":"ContainerDied","Data":"5eb9b8e091aacf458d76fc956c10d7d3d453d7a64f338e5d464d212f737489dc"}
Jul 08 16:55:47.427338 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:47.427232    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz" event={"ID":"59595c67-9459-4614-8e29-836f7139515b","Type":"ContainerDied","Data":"fd0d163e0ecebffcd300b42cd3997ba96d0b06520ed99ea89d1d08b9c7378994"}
Jul 08 16:55:47.427338 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:47.427244    2565 scope.go:117] "RemoveContainer" containerID="5eb9b8e091aacf458d76fc956c10d7d3d453d7a64f338e5d464d212f737489dc"
Jul 08 16:55:47.444385 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:47.444361    2565 scope.go:117] "RemoveContainer" containerID="e3bcf3b4e4794c8ad03bef48362c04834c103d7042d99c0781a48f4c1533cfc1"
Jul 08 16:55:47.454443 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:47.454413    2565 scope.go:117] "RemoveContainer" containerID="5eb9b8e091aacf458d76fc956c10d7d3d453d7a64f338e5d464d212f737489dc"
Jul 08 16:55:47.454963 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:55:47.454933    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eb9b8e091aacf458d76fc956c10d7d3d453d7a64f338e5d464d212f737489dc\": container with ID starting with 5eb9b8e091aacf458d76fc956c10d7d3d453d7a64f338e5d464d212f737489dc not found: ID does not exist" containerID="5eb9b8e091aacf458d76fc956c10d7d3d453d7a64f338e5d464d212f737489dc"
Jul 08 16:55:47.455085 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:47.454975    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eb9b8e091aacf458d76fc956c10d7d3d453d7a64f338e5d464d212f737489dc"} err="failed to get container status \"5eb9b8e091aacf458d76fc956c10d7d3d453d7a64f338e5d464d212f737489dc\": rpc error: code = NotFound desc = could not find container \"5eb9b8e091aacf458d76fc956c10d7d3d453d7a64f338e5d464d212f737489dc\": container with ID starting with 5eb9b8e091aacf458d76fc956c10d7d3d453d7a64f338e5d464d212f737489dc not found: ID does not exist"
Jul 08 16:55:47.455085 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:47.455002    2565 scope.go:117] "RemoveContainer" containerID="e3bcf3b4e4794c8ad03bef48362c04834c103d7042d99c0781a48f4c1533cfc1"
Jul 08 16:55:47.455354 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:55:47.455333    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3bcf3b4e4794c8ad03bef48362c04834c103d7042d99c0781a48f4c1533cfc1\": container with ID starting with e3bcf3b4e4794c8ad03bef48362c04834c103d7042d99c0781a48f4c1533cfc1 not found: ID does not exist" containerID="e3bcf3b4e4794c8ad03bef48362c04834c103d7042d99c0781a48f4c1533cfc1"
Jul 08 16:55:47.455417 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:47.455359    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3bcf3b4e4794c8ad03bef48362c04834c103d7042d99c0781a48f4c1533cfc1"} err="failed to get container status \"e3bcf3b4e4794c8ad03bef48362c04834c103d7042d99c0781a48f4c1533cfc1\": rpc error: code = NotFound desc = could not find container \"e3bcf3b4e4794c8ad03bef48362c04834c103d7042d99c0781a48f4c1533cfc1\": container with ID starting with e3bcf3b4e4794c8ad03bef48362c04834c103d7042d99c0781a48f4c1533cfc1 not found: ID does not exist"
Jul 08 16:55:47.460052 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:47.460019    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz"]
Jul 08 16:55:47.466134 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:47.466100    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-router-scheduler-7575d494fd-whhqz"]
Jul 08 16:55:47.619103 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:47.619063    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986" podUID="5d5691ff-d2c2-4559-82de-70d5dccdf797" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 08 16:55:47.822965 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:47.822867    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59595c67-9459-4614-8e29-836f7139515b" path="/var/lib/kubelet/pods/59595c67-9459-4614-8e29-836f7139515b/volumes"
Jul 08 16:55:57.619307 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:55:57.619265    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986" podUID="5d5691ff-d2c2-4559-82de-70d5dccdf797" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 08 16:56:01.466379 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:01.466316    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4" podUID="97052463-d959-4ece-8c21-5d267ccdebf2" containerName="main" containerID="cri-o://3cd7f8ca6a43ee712d823c1f294f00b909b97b7feddb1f3e4c7ec306251e8285" gracePeriod=45
Jul 08 16:56:01.855401 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:01.855289    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4" podUID="97052463-d959-4ece-8c21-5d267ccdebf2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.53:8000/health\": dial tcp 10.133.0.53:8000: connect: connection refused"
Jul 08 16:56:02.225260 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.225233    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"
Jul 08 16:56:02.319615 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.319583    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-tmp-dir\") pod \"97052463-d959-4ece-8c21-5d267ccdebf2\" (UID: \"97052463-d959-4ece-8c21-5d267ccdebf2\") "
Jul 08 16:56:02.319615 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.319622    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ph9nj\" (UniqueName: \"kubernetes.io/projected/97052463-d959-4ece-8c21-5d267ccdebf2-kube-api-access-ph9nj\") pod \"97052463-d959-4ece-8c21-5d267ccdebf2\" (UID: \"97052463-d959-4ece-8c21-5d267ccdebf2\") "
Jul 08 16:56:02.319925 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.319639    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-model-cache\") pod \"97052463-d959-4ece-8c21-5d267ccdebf2\" (UID: \"97052463-d959-4ece-8c21-5d267ccdebf2\") "
Jul 08 16:56:02.319925 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.319660    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-kserve-provision-location\") pod \"97052463-d959-4ece-8c21-5d267ccdebf2\" (UID: \"97052463-d959-4ece-8c21-5d267ccdebf2\") "
Jul 08 16:56:02.319925 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.319725    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-dshm\") pod \"97052463-d959-4ece-8c21-5d267ccdebf2\" (UID: \"97052463-d959-4ece-8c21-5d267ccdebf2\") "
Jul 08 16:56:02.319925 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.319804    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-home\") pod \"97052463-d959-4ece-8c21-5d267ccdebf2\" (UID: \"97052463-d959-4ece-8c21-5d267ccdebf2\") "
Jul 08 16:56:02.319925 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.319836    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97052463-d959-4ece-8c21-5d267ccdebf2-tls-certs\") pod \"97052463-d959-4ece-8c21-5d267ccdebf2\" (UID: \"97052463-d959-4ece-8c21-5d267ccdebf2\") "
Jul 08 16:56:02.320253 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.319936    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-model-cache" (OuterVolumeSpecName: "model-cache") pod "97052463-d959-4ece-8c21-5d267ccdebf2" (UID: "97052463-d959-4ece-8c21-5d267ccdebf2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:56:02.320253 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.320133    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:56:02.320649 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.320615    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-home" (OuterVolumeSpecName: "home") pod "97052463-d959-4ece-8c21-5d267ccdebf2" (UID: "97052463-d959-4ece-8c21-5d267ccdebf2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:56:02.322033 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.321997    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97052463-d959-4ece-8c21-5d267ccdebf2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "97052463-d959-4ece-8c21-5d267ccdebf2" (UID: "97052463-d959-4ece-8c21-5d267ccdebf2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:56:02.322129 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.322086    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-dshm" (OuterVolumeSpecName: "dshm") pod "97052463-d959-4ece-8c21-5d267ccdebf2" (UID: "97052463-d959-4ece-8c21-5d267ccdebf2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:56:02.322363 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.322338    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97052463-d959-4ece-8c21-5d267ccdebf2-kube-api-access-ph9nj" (OuterVolumeSpecName: "kube-api-access-ph9nj") pod "97052463-d959-4ece-8c21-5d267ccdebf2" (UID: "97052463-d959-4ece-8c21-5d267ccdebf2"). InnerVolumeSpecName "kube-api-access-ph9nj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:56:02.334602 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.334557    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "97052463-d959-4ece-8c21-5d267ccdebf2" (UID: "97052463-d959-4ece-8c21-5d267ccdebf2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:56:02.397116 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.397034    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "97052463-d959-4ece-8c21-5d267ccdebf2" (UID: "97052463-d959-4ece-8c21-5d267ccdebf2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:56:02.420841 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.420806    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph9nj\" (UniqueName: \"kubernetes.io/projected/97052463-d959-4ece-8c21-5d267ccdebf2-kube-api-access-ph9nj\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:56:02.420841 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.420835    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-kserve-provision-location\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:56:02.420841 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.420849    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:56:02.421075 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.420858    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:56:02.421075 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.420869    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97052463-d959-4ece-8c21-5d267ccdebf2-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:56:02.421075 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.420879    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97052463-d959-4ece-8c21-5d267ccdebf2-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:56:02.519404 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.519367    2565 generic.go:356] "Generic (PLEG): container finished" podID="97052463-d959-4ece-8c21-5d267ccdebf2" containerID="3cd7f8ca6a43ee712d823c1f294f00b909b97b7feddb1f3e4c7ec306251e8285" exitCode=0
Jul 08 16:56:02.519847 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.519415    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4" event={"ID":"97052463-d959-4ece-8c21-5d267ccdebf2","Type":"ContainerDied","Data":"3cd7f8ca6a43ee712d823c1f294f00b909b97b7feddb1f3e4c7ec306251e8285"}
Jul 08 16:56:02.519847 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.519474    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4" event={"ID":"97052463-d959-4ece-8c21-5d267ccdebf2","Type":"ContainerDied","Data":"2709d6ebad52bc7ad6729daf4a08f789f6fbaec2a04ef1c0b8111dbc4ded0aa7"}
Jul 08 16:56:02.519847 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.519498    2565 scope.go:117] "RemoveContainer" containerID="3cd7f8ca6a43ee712d823c1f294f00b909b97b7feddb1f3e4c7ec306251e8285"
Jul 08 16:56:02.519847 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.519427    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"
Jul 08 16:56:02.529718 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.529694    2565 scope.go:117] "RemoveContainer" containerID="3259a4d7dc041e28be39d76e5d377f287f1183e0453f1af416ff867c13c87f42"
Jul 08 16:56:02.540591 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.540569    2565 scope.go:117] "RemoveContainer" containerID="3cd7f8ca6a43ee712d823c1f294f00b909b97b7feddb1f3e4c7ec306251e8285"
Jul 08 16:56:02.540920 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:56:02.540900    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cd7f8ca6a43ee712d823c1f294f00b909b97b7feddb1f3e4c7ec306251e8285\": container with ID starting with 3cd7f8ca6a43ee712d823c1f294f00b909b97b7feddb1f3e4c7ec306251e8285 not found: ID does not exist" containerID="3cd7f8ca6a43ee712d823c1f294f00b909b97b7feddb1f3e4c7ec306251e8285"
Jul 08 16:56:02.540971 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.540936    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cd7f8ca6a43ee712d823c1f294f00b909b97b7feddb1f3e4c7ec306251e8285"} err="failed to get container status \"3cd7f8ca6a43ee712d823c1f294f00b909b97b7feddb1f3e4c7ec306251e8285\": rpc error: code = NotFound desc = could not find container \"3cd7f8ca6a43ee712d823c1f294f00b909b97b7feddb1f3e4c7ec306251e8285\": container with ID starting with 3cd7f8ca6a43ee712d823c1f294f00b909b97b7feddb1f3e4c7ec306251e8285 not found: ID does not exist"
Jul 08 16:56:02.540971 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.540956    2565 scope.go:117] "RemoveContainer" containerID="3259a4d7dc041e28be39d76e5d377f287f1183e0453f1af416ff867c13c87f42"
Jul 08 16:56:02.541205 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:56:02.541184    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3259a4d7dc041e28be39d76e5d377f287f1183e0453f1af416ff867c13c87f42\": container with ID starting with 3259a4d7dc041e28be39d76e5d377f287f1183e0453f1af416ff867c13c87f42 not found: ID does not exist" containerID="3259a4d7dc041e28be39d76e5d377f287f1183e0453f1af416ff867c13c87f42"
Jul 08 16:56:02.541299 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.541209    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3259a4d7dc041e28be39d76e5d377f287f1183e0453f1af416ff867c13c87f42"} err="failed to get container status \"3259a4d7dc041e28be39d76e5d377f287f1183e0453f1af416ff867c13c87f42\": rpc error: code = NotFound desc = could not find container \"3259a4d7dc041e28be39d76e5d377f287f1183e0453f1af416ff867c13c87f42\": container with ID starting with 3259a4d7dc041e28be39d76e5d377f287f1183e0453f1af416ff867c13c87f42 not found: ID does not exist"
Jul 08 16:56:02.546963 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.546933    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"]
Jul 08 16:56:02.554871 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:02.554842    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/auth-disabled-test-kserve-84ff994f99-sl6v4"]
Jul 08 16:56:03.830507 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:03.830475    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97052463-d959-4ece-8c21-5d267ccdebf2" path="/var/lib/kubelet/pods/97052463-d959-4ece-8c21-5d267ccdebf2/volumes"
Jul 08 16:56:07.618593 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:07.618555    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986" podUID="5d5691ff-d2c2-4559-82de-70d5dccdf797" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 08 16:56:15.416114 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:15.416072    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:56:17.619077 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:17.619030    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986" podUID="5d5691ff-d2c2-4559-82de-70d5dccdf797" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 08 16:56:23.604891 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:23.604843    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:56:25.683408 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:25.683370    2565 generic.go:356] "Generic (PLEG): container finished" podID="279f6cb5-9686-4583-b06c-bd27b344f0a9" containerID="d60f02af33b7ee0dc857f05f26da85d405765c74fe4402efa098edee7e914f1c" exitCode=0
Jul 08 16:56:25.683979 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:25.683425    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz" event={"ID":"279f6cb5-9686-4583-b06c-bd27b344f0a9","Type":"ContainerDied","Data":"d60f02af33b7ee0dc857f05f26da85d405765c74fe4402efa098edee7e914f1c"}
Jul 08 16:56:25.684054 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:25.683979    2565 scope.go:117] "RemoveContainer" containerID="d60f02af33b7ee0dc857f05f26da85d405765c74fe4402efa098edee7e914f1c"
Jul 08 16:56:25.685025 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:25.684987    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 08 16:56:25.685124 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:25.685063    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 08 16:56:25.685124 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:25.685097    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 08 16:56:26.692230 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:26.692191    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz" event={"ID":"279f6cb5-9686-4583-b06c-bd27b344f0a9","Type":"ContainerStarted","Data":"8b58a929c39171daf247d53d87f5cc8e16f280c73dd5c299c456431877bd8e9d"}
Jul 08 16:56:26.734246 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:26.734204    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:56:26.735755 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:26.735715    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz" podUID="279f6cb5-9686-4583-b06c-bd27b344f0a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Jul 08 16:56:27.618915 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:27.618874    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986" podUID="5d5691ff-d2c2-4559-82de-70d5dccdf797" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 08 16:56:27.734245 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:27.734190    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz" podUID="279f6cb5-9686-4583-b06c-bd27b344f0a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Jul 08 16:56:28.042763 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:28.042721    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:56:28.733938 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:28.733895    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz" podUID="279f6cb5-9686-4583-b06c-bd27b344f0a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Jul 08 16:56:29.733861 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:29.733808    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz" podUID="279f6cb5-9686-4583-b06c-bd27b344f0a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Jul 08 16:56:30.734599 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:30.734530    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz" podUID="279f6cb5-9686-4583-b06c-bd27b344f0a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Jul 08 16:56:31.733781 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:31.733734    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz" podUID="279f6cb5-9686-4583-b06c-bd27b344f0a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused"
Jul 08 16:56:32.738494 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:32.738461    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:56:32.739504 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:32.739481    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-1-openshift-default-54c47cdfc7-rxfpz"
Jul 08 16:56:37.618871 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:37.618817    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986" podUID="5d5691ff-d2c2-4559-82de-70d5dccdf797" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 08 16:56:47.619383 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:47.619284    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986" podUID="5d5691ff-d2c2-4559-82de-70d5dccdf797" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": dial tcp 10.133.0.55:8000: connect: connection refused"
Jul 08 16:56:55.107572 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:55.107527    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:56:57.628302 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:57.628273    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:56:57.636425 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:56:57.636391    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:57:23.915564 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:23.915529    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"]
Jul 08 16:57:23.929378 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:23.929343    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:57:24.029939 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:24.029908    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:57:27.105291 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:27.105250    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:57:38.939897 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:38.939853    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986" podUID="5d5691ff-d2c2-4559-82de-70d5dccdf797" containerName="main" containerID="cri-o://d02da8a4f791f4edae66cf9244a39610a37542252d1ac3b8e409f856ab3888b5" gracePeriod=45
Jul 08 16:57:39.590005 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:39.589980    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:57:39.767630 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:39.767587    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwpn9\" (UniqueName: \"kubernetes.io/projected/5d5691ff-d2c2-4559-82de-70d5dccdf797-kube-api-access-hwpn9\") pod \"5d5691ff-d2c2-4559-82de-70d5dccdf797\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") "
Jul 08 16:57:39.767630 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:39.767630    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-home\") pod \"5d5691ff-d2c2-4559-82de-70d5dccdf797\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") "
Jul 08 16:57:39.767932 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:39.767669    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-tmp-dir\") pod \"5d5691ff-d2c2-4559-82de-70d5dccdf797\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") "
Jul 08 16:57:39.767932 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:39.767780    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d5691ff-d2c2-4559-82de-70d5dccdf797-tls-certs\") pod \"5d5691ff-d2c2-4559-82de-70d5dccdf797\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") "
Jul 08 16:57:39.767932 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:39.767798    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d5691ff-d2c2-4559-82de-70d5dccdf797-cabundle-cert\") pod \"5d5691ff-d2c2-4559-82de-70d5dccdf797\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") "
Jul 08 16:57:39.767932 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:39.767815    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-dshm\") pod \"5d5691ff-d2c2-4559-82de-70d5dccdf797\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") "
Jul 08 16:57:39.767932 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:39.767832    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-model-cache\") pod \"5d5691ff-d2c2-4559-82de-70d5dccdf797\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") "
Jul 08 16:57:39.767932 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:39.767847    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-kserve-provision-location\") pod \"5d5691ff-d2c2-4559-82de-70d5dccdf797\" (UID: \"5d5691ff-d2c2-4559-82de-70d5dccdf797\") "
Jul 08 16:57:39.768250 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:39.768196    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d5691ff-d2c2-4559-82de-70d5dccdf797-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "5d5691ff-d2c2-4559-82de-70d5dccdf797" (UID: "5d5691ff-d2c2-4559-82de-70d5dccdf797"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:57:39.768250 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:39.768209    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-home" (OuterVolumeSpecName: "home") pod "5d5691ff-d2c2-4559-82de-70d5dccdf797" (UID: "5d5691ff-d2c2-4559-82de-70d5dccdf797"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:57:39.768366 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:39.768344    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-model-cache" (OuterVolumeSpecName: "model-cache") pod "5d5691ff-d2c2-4559-82de-70d5dccdf797" (UID: "5d5691ff-d2c2-4559-82de-70d5dccdf797"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:57:39.770136 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:39.770109    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-dshm" (OuterVolumeSpecName: "dshm") pod "5d5691ff-d2c2-4559-82de-70d5dccdf797" (UID: "5d5691ff-d2c2-4559-82de-70d5dccdf797"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:57:39.770252 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:39.770195    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d5691ff-d2c2-4559-82de-70d5dccdf797-kube-api-access-hwpn9" (OuterVolumeSpecName: "kube-api-access-hwpn9") pod "5d5691ff-d2c2-4559-82de-70d5dccdf797" (UID: "5d5691ff-d2c2-4559-82de-70d5dccdf797"). InnerVolumeSpecName "kube-api-access-hwpn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:57:39.770345 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:39.770328    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d5691ff-d2c2-4559-82de-70d5dccdf797-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5d5691ff-d2c2-4559-82de-70d5dccdf797" (UID: "5d5691ff-d2c2-4559-82de-70d5dccdf797"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:57:39.779988 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:39.779965    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5d5691ff-d2c2-4559-82de-70d5dccdf797" (UID: "5d5691ff-d2c2-4559-82de-70d5dccdf797"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:57:39.836247 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:39.836202    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "5d5691ff-d2c2-4559-82de-70d5dccdf797" (UID: "5d5691ff-d2c2-4559-82de-70d5dccdf797"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:57:39.869218 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:39.869182    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:57:39.869218 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:39.869214    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d5691ff-d2c2-4559-82de-70d5dccdf797-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:57:39.869218 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:39.869224    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/5d5691ff-d2c2-4559-82de-70d5dccdf797-cabundle-cert\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:57:39.869418 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:39.869233    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:57:39.869418 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:39.869241    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:57:39.869418 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:39.869251    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-kserve-provision-location\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:57:39.869418 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:39.869259    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwpn9\" (UniqueName: \"kubernetes.io/projected/5d5691ff-d2c2-4559-82de-70d5dccdf797-kube-api-access-hwpn9\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:57:39.869418 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:39.869267    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d5691ff-d2c2-4559-82de-70d5dccdf797-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:57:40.124147 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:40.124056    2565 generic.go:356] "Generic (PLEG): container finished" podID="5d5691ff-d2c2-4559-82de-70d5dccdf797" containerID="d02da8a4f791f4edae66cf9244a39610a37542252d1ac3b8e409f856ab3888b5" exitCode=0
Jul 08 16:57:40.124147 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:40.124136    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"
Jul 08 16:57:40.124613 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:40.124140    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986" event={"ID":"5d5691ff-d2c2-4559-82de-70d5dccdf797","Type":"ContainerDied","Data":"d02da8a4f791f4edae66cf9244a39610a37542252d1ac3b8e409f856ab3888b5"}
Jul 08 16:57:40.124613 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:40.124197    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986" event={"ID":"5d5691ff-d2c2-4559-82de-70d5dccdf797","Type":"ContainerDied","Data":"855cb77f24fc9a34cac901073e2cbc9b3c34a0a269d1b6d6793af5f0309d453b"}
Jul 08 16:57:40.124613 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:40.124220    2565 scope.go:117] "RemoveContainer" containerID="d02da8a4f791f4edae66cf9244a39610a37542252d1ac3b8e409f856ab3888b5"
Jul 08 16:57:40.133775 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:40.133749    2565 scope.go:117] "RemoveContainer" containerID="831c3cb341971c18ff935c13122ef81216044f3f38993abc36ff00f44fafaae0"
Jul 08 16:57:40.142471 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:40.142447    2565 scope.go:117] "RemoveContainer" containerID="d02da8a4f791f4edae66cf9244a39610a37542252d1ac3b8e409f856ab3888b5"
Jul 08 16:57:40.142776 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:57:40.142755    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d02da8a4f791f4edae66cf9244a39610a37542252d1ac3b8e409f856ab3888b5\": container with ID starting with d02da8a4f791f4edae66cf9244a39610a37542252d1ac3b8e409f856ab3888b5 not found: ID does not exist" containerID="d02da8a4f791f4edae66cf9244a39610a37542252d1ac3b8e409f856ab3888b5"
Jul 08 16:57:40.142850 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:40.142785    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d02da8a4f791f4edae66cf9244a39610a37542252d1ac3b8e409f856ab3888b5"} err="failed to get container status \"d02da8a4f791f4edae66cf9244a39610a37542252d1ac3b8e409f856ab3888b5\": rpc error: code = NotFound desc = could not find container \"d02da8a4f791f4edae66cf9244a39610a37542252d1ac3b8e409f856ab3888b5\": container with ID starting with d02da8a4f791f4edae66cf9244a39610a37542252d1ac3b8e409f856ab3888b5 not found: ID does not exist"
Jul 08 16:57:40.142850 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:40.142804    2565 scope.go:117] "RemoveContainer" containerID="831c3cb341971c18ff935c13122ef81216044f3f38993abc36ff00f44fafaae0"
Jul 08 16:57:40.143036 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:57:40.143014    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"831c3cb341971c18ff935c13122ef81216044f3f38993abc36ff00f44fafaae0\": container with ID starting with 831c3cb341971c18ff935c13122ef81216044f3f38993abc36ff00f44fafaae0 not found: ID does not exist" containerID="831c3cb341971c18ff935c13122ef81216044f3f38993abc36ff00f44fafaae0"
Jul 08 16:57:40.143098 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:40.143047    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"831c3cb341971c18ff935c13122ef81216044f3f38993abc36ff00f44fafaae0"} err="failed to get container status \"831c3cb341971c18ff935c13122ef81216044f3f38993abc36ff00f44fafaae0\": rpc error: code = NotFound desc = could not find container \"831c3cb341971c18ff935c13122ef81216044f3f38993abc36ff00f44fafaae0\": container with ID starting with 831c3cb341971c18ff935c13122ef81216044f3f38993abc36ff00f44fafaae0 not found: ID does not exist"
Jul 08 16:57:40.150274 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:40.150251    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"]
Jul 08 16:57:40.156437 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:40.156416    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-7ca60146-kserve-6c94dd65d4-jd986"]
Jul 08 16:57:41.822586 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:41.822549    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d5691ff-d2c2-4559-82de-70d5dccdf797" path="/var/lib/kubelet/pods/5d5691ff-d2c2-4559-82de-70d5dccdf797/volumes"
Jul 08 16:57:56.692744 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.692705    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"]
Jul 08 16:57:56.693202 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.693072    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97052463-d959-4ece-8c21-5d267ccdebf2" containerName="storage-initializer"
Jul 08 16:57:56.693202 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.693083    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="97052463-d959-4ece-8c21-5d267ccdebf2" containerName="storage-initializer"
Jul 08 16:57:56.693202 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.693095    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d5691ff-d2c2-4559-82de-70d5dccdf797" containerName="main"
Jul 08 16:57:56.693202 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.693100    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d5691ff-d2c2-4559-82de-70d5dccdf797" containerName="main"
Jul 08 16:57:56.693202 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.693108    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59595c67-9459-4614-8e29-836f7139515b" containerName="main"
Jul 08 16:57:56.693202 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.693114    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="59595c67-9459-4614-8e29-836f7139515b" containerName="main"
Jul 08 16:57:56.693202 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.693125    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59595c67-9459-4614-8e29-836f7139515b" containerName="storage-initializer"
Jul 08 16:57:56.693202 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.693130    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="59595c67-9459-4614-8e29-836f7139515b" containerName="storage-initializer"
Jul 08 16:57:56.693202 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.693136    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97052463-d959-4ece-8c21-5d267ccdebf2" containerName="main"
Jul 08 16:57:56.693202 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.693141    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="97052463-d959-4ece-8c21-5d267ccdebf2" containerName="main"
Jul 08 16:57:56.693202 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.693148    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d5691ff-d2c2-4559-82de-70d5dccdf797" containerName="storage-initializer"
Jul 08 16:57:56.693202 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.693152    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d5691ff-d2c2-4559-82de-70d5dccdf797" containerName="storage-initializer"
Jul 08 16:57:56.693618 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.693210    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="97052463-d959-4ece-8c21-5d267ccdebf2" containerName="main"
Jul 08 16:57:56.693618 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.693219    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d5691ff-d2c2-4559-82de-70d5dccdf797" containerName="main"
Jul 08 16:57:56.693618 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.693227    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="59595c67-9459-4614-8e29-836f7139515b" containerName="main"
Jul 08 16:57:56.696375 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.696355    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:57:56.699059 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.699036    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qxdr\""
Jul 08 16:57:56.699188 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.699039    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 16:57:56.699188 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.699111    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv77ff2528d3e9b4972cd9335229fce9f0-kserve-self-signed-certs\""
Jul 08 16:57:56.699188 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.699169    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 08 16:57:56.721199 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.721173    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"]
Jul 08 16:57:56.740949 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.740916    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:57:56.812242 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.812210    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:57:56.812460 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.812246    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:57:56.812460 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.812279    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:57:56.812460 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.812296    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:57:56.812460 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.812375    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:57:56.812460 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.812411    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkq4b\" (UniqueName: \"kubernetes.io/projected/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-kube-api-access-fkq4b\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:57:56.812750 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.812529    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:57:56.812750 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.812562    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:57:56.913213 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.913181    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:57:56.913213 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.913216    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:57:56.913469 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.913350    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:57:56.913469 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.913391    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:57:56.913469 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.913446    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:57:56.913626 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.913473    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:57:56.913626 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.913538    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:57:56.913626 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.913538    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:57:56.913626 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.913589    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkq4b\" (UniqueName: \"kubernetes.io/projected/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-kube-api-access-fkq4b\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:57:56.913877 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.913666    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-home\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:57:56.913877 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.913771    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-model-cache\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:57:56.913975 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.913878    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:57:56.914210 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.914186    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-cabundle-cert\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:57:56.915607 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.915585    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-dshm\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:57:56.915997 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.915974    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-tls-certs\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:57:56.923104 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:56.923073    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkq4b\" (UniqueName: \"kubernetes.io/projected/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-kube-api-access-fkq4b\") pod \"llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:57:57.007452 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:57.007358    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:57:57.160701 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:57.160656    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"]
Jul 08 16:57:57.162250 ip-10-0-128-142 kubenswrapper[2565]: W0708 16:57:57.162227    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd5979dd_a171_4f01_adf7_da2f7ffd5a0c.slice/crio-18e679aac1f85778160106e4091fdc9b55d3f43723286e81128ae2c56a50f129 WatchSource:0}: Error finding container 18e679aac1f85778160106e4091fdc9b55d3f43723286e81128ae2c56a50f129: Status 404 returned error can't find the container with id 18e679aac1f85778160106e4091fdc9b55d3f43723286e81128ae2c56a50f129
Jul 08 16:57:57.183278 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:57.183248    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:57:57.221941 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:57.221915    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n" event={"ID":"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c","Type":"ContainerStarted","Data":"18e679aac1f85778160106e4091fdc9b55d3f43723286e81128ae2c56a50f129"}
Jul 08 16:57:58.230330 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:57:58.230290    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n" event={"ID":"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c","Type":"ContainerStarted","Data":"8c1c8beee6fd3da4a3f6247bb0cd2a645109f32f6cbbb843b9adbcdae71a1527"}
Jul 08 16:58:11.642569 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:58:11.642533    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:58:21.365173 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:58:21.365139    2565 generic.go:356] "Generic (PLEG): container finished" podID="cd5979dd-a171-4f01-adf7-da2f7ffd5a0c" containerID="8c1c8beee6fd3da4a3f6247bb0cd2a645109f32f6cbbb843b9adbcdae71a1527" exitCode=0
Jul 08 16:58:21.365599 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:58:21.365212    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n" event={"ID":"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c","Type":"ContainerDied","Data":"8c1c8beee6fd3da4a3f6247bb0cd2a645109f32f6cbbb843b9adbcdae71a1527"}
Jul 08 16:58:22.375769 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:58:22.375731    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n" event={"ID":"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c","Type":"ContainerStarted","Data":"888e9152b0cf0815591351a3f8b05fa44793fa325ad0d8db69a2a0f3cb1f263d"}
Jul 08 16:58:22.403187 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:58:22.403138    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n" podStartSLOduration=2.352718406 podStartE2EDuration="26.40312077s" totalImagesPullingTime="0s" totalInitContainerRuntime="24.050402364s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:58:22.399928467 +0000 UTC m=+1439.113289911" watchObservedRunningTime="2026-07-08 16:58:22.40312077 +0000 UTC m=+1439.116482196"
Jul 08 16:58:23.007738 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:58:23.007701    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:58:27.008062 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:58:27.008026    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:58:27.009780 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:58:27.009751    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n" podUID="cd5979dd-a171-4f01-adf7-da2f7ffd5a0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 08 16:58:37.008589 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:58:37.008543    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n" podUID="cd5979dd-a171-4f01-adf7-da2f7ffd5a0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 08 16:58:47.008691 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:58:47.008639    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n" podUID="cd5979dd-a171-4f01-adf7-da2f7ffd5a0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 08 16:58:57.008091 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:58:57.008047    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n" podUID="cd5979dd-a171-4f01-adf7-da2f7ffd5a0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 08 16:59:07.008441 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:07.008391    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n" podUID="cd5979dd-a171-4f01-adf7-da2f7ffd5a0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 08 16:59:17.008651 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:17.008605    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n" podUID="cd5979dd-a171-4f01-adf7-da2f7ffd5a0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 08 16:59:27.008334 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:27.008279    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n" podUID="cd5979dd-a171-4f01-adf7-da2f7ffd5a0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 08 16:59:37.018719 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:37.018662    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:59:37.026691 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:37.026657    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:59:38.885204 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:38.885172    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"]
Jul 08 16:59:38.901374 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:38.901343    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:59:38.983970 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:38.983931    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 16:59:53.908931 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:53.908824    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n" podUID="cd5979dd-a171-4f01-adf7-da2f7ffd5a0c" containerName="main" containerID="cri-o://888e9152b0cf0815591351a3f8b05fa44793fa325ad0d8db69a2a0f3cb1f263d" gracePeriod=45
Jul 08 16:59:54.020022 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.019981    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n" podUID="cd5979dd-a171-4f01-adf7-da2f7ffd5a0c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/health\": dial tcp 10.133.0.56:8000: connect: connection refused"
Jul 08 16:59:54.468345 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.468319    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:59:54.573775 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.573669    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-kserve-provision-location\") pod \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") "
Jul 08 16:59:54.573775 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.573746    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-tls-certs\") pod \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") "
Jul 08 16:59:54.573775 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.573765    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkq4b\" (UniqueName: \"kubernetes.io/projected/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-kube-api-access-fkq4b\") pod \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") "
Jul 08 16:59:54.574062 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.573782    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-cabundle-cert\") pod \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") "
Jul 08 16:59:54.574062 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.573813    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-home\") pod \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") "
Jul 08 16:59:54.574062 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.573832    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-model-cache\") pod \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") "
Jul 08 16:59:54.574062 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.573867    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-dshm\") pod \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") "
Jul 08 16:59:54.574062 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.573899    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-tmp-dir\") pod \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\" (UID: \"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c\") "
Jul 08 16:59:54.574320 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.574108    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-model-cache" (OuterVolumeSpecName: "model-cache") pod "cd5979dd-a171-4f01-adf7-da2f7ffd5a0c" (UID: "cd5979dd-a171-4f01-adf7-da2f7ffd5a0c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:59:54.574320 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.574238    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:59:54.574439 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.574254    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-home" (OuterVolumeSpecName: "home") pod "cd5979dd-a171-4f01-adf7-da2f7ffd5a0c" (UID: "cd5979dd-a171-4f01-adf7-da2f7ffd5a0c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:59:54.574439 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.574362    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cd5979dd-a171-4f01-adf7-da2f7ffd5a0c" (UID: "cd5979dd-a171-4f01-adf7-da2f7ffd5a0c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 16:59:54.574439 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.574420    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "cd5979dd-a171-4f01-adf7-da2f7ffd5a0c" (UID: "cd5979dd-a171-4f01-adf7-da2f7ffd5a0c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:59:54.576003 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.575979    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-kube-api-access-fkq4b" (OuterVolumeSpecName: "kube-api-access-fkq4b") pod "cd5979dd-a171-4f01-adf7-da2f7ffd5a0c" (UID: "cd5979dd-a171-4f01-adf7-da2f7ffd5a0c"). InnerVolumeSpecName "kube-api-access-fkq4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 16:59:54.576182 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.576021    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "cd5979dd-a171-4f01-adf7-da2f7ffd5a0c" (UID: "cd5979dd-a171-4f01-adf7-da2f7ffd5a0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 16:59:54.576182 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.576101    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-dshm" (OuterVolumeSpecName: "dshm") pod "cd5979dd-a171-4f01-adf7-da2f7ffd5a0c" (UID: "cd5979dd-a171-4f01-adf7-da2f7ffd5a0c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:59:54.638913 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.638876    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cd5979dd-a171-4f01-adf7-da2f7ffd5a0c" (UID: "cd5979dd-a171-4f01-adf7-da2f7ffd5a0c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 16:59:54.674897 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.674873    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:59:54.674897 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.674893    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:59:54.675047 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.674922    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:59:54.675047 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.674936    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-kserve-provision-location\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:59:54.675047 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.674944    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:59:54.675047 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.674953    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkq4b\" (UniqueName: \"kubernetes.io/projected/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-kube-api-access-fkq4b\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:59:54.675047 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.674961    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c-cabundle-cert\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 16:59:54.919542 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.919458    2565 generic.go:356] "Generic (PLEG): container finished" podID="cd5979dd-a171-4f01-adf7-da2f7ffd5a0c" containerID="888e9152b0cf0815591351a3f8b05fa44793fa325ad0d8db69a2a0f3cb1f263d" exitCode=0
Jul 08 16:59:54.919542 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.919517    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"
Jul 08 16:59:54.919542 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.919520    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n" event={"ID":"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c","Type":"ContainerDied","Data":"888e9152b0cf0815591351a3f8b05fa44793fa325ad0d8db69a2a0f3cb1f263d"}
Jul 08 16:59:54.920124 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.919572    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n" event={"ID":"cd5979dd-a171-4f01-adf7-da2f7ffd5a0c","Type":"ContainerDied","Data":"18e679aac1f85778160106e4091fdc9b55d3f43723286e81128ae2c56a50f129"}
Jul 08 16:59:54.920124 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.919603    2565 scope.go:117] "RemoveContainer" containerID="888e9152b0cf0815591351a3f8b05fa44793fa325ad0d8db69a2a0f3cb1f263d"
Jul 08 16:59:54.929053 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.929035    2565 scope.go:117] "RemoveContainer" containerID="8c1c8beee6fd3da4a3f6247bb0cd2a645109f32f6cbbb843b9adbcdae71a1527"
Jul 08 16:59:54.937288 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.937273    2565 scope.go:117] "RemoveContainer" containerID="888e9152b0cf0815591351a3f8b05fa44793fa325ad0d8db69a2a0f3cb1f263d"
Jul 08 16:59:54.937515 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:59:54.937499    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"888e9152b0cf0815591351a3f8b05fa44793fa325ad0d8db69a2a0f3cb1f263d\": container with ID starting with 888e9152b0cf0815591351a3f8b05fa44793fa325ad0d8db69a2a0f3cb1f263d not found: ID does not exist" containerID="888e9152b0cf0815591351a3f8b05fa44793fa325ad0d8db69a2a0f3cb1f263d"
Jul 08 16:59:54.937559 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.937524    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"888e9152b0cf0815591351a3f8b05fa44793fa325ad0d8db69a2a0f3cb1f263d"} err="failed to get container status \"888e9152b0cf0815591351a3f8b05fa44793fa325ad0d8db69a2a0f3cb1f263d\": rpc error: code = NotFound desc = could not find container \"888e9152b0cf0815591351a3f8b05fa44793fa325ad0d8db69a2a0f3cb1f263d\": container with ID starting with 888e9152b0cf0815591351a3f8b05fa44793fa325ad0d8db69a2a0f3cb1f263d not found: ID does not exist"
Jul 08 16:59:54.937559 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.937538    2565 scope.go:117] "RemoveContainer" containerID="8c1c8beee6fd3da4a3f6247bb0cd2a645109f32f6cbbb843b9adbcdae71a1527"
Jul 08 16:59:54.937759 ip-10-0-128-142 kubenswrapper[2565]: E0708 16:59:54.937741    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c1c8beee6fd3da4a3f6247bb0cd2a645109f32f6cbbb843b9adbcdae71a1527\": container with ID starting with 8c1c8beee6fd3da4a3f6247bb0cd2a645109f32f6cbbb843b9adbcdae71a1527 not found: ID does not exist" containerID="8c1c8beee6fd3da4a3f6247bb0cd2a645109f32f6cbbb843b9adbcdae71a1527"
Jul 08 16:59:54.937814 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.937767    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c1c8beee6fd3da4a3f6247bb0cd2a645109f32f6cbbb843b9adbcdae71a1527"} err="failed to get container status \"8c1c8beee6fd3da4a3f6247bb0cd2a645109f32f6cbbb843b9adbcdae71a1527\": rpc error: code = NotFound desc = could not find container \"8c1c8beee6fd3da4a3f6247bb0cd2a645109f32f6cbbb843b9adbcdae71a1527\": container with ID starting with 8c1c8beee6fd3da4a3f6247bb0cd2a645109f32f6cbbb843b9adbcdae71a1527 not found: ID does not exist"
Jul 08 16:59:54.949582 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.949557    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"]
Jul 08 16:59:54.952593 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:54.952572    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-with-ba4d693a-kserve-7dfc79f5dc-wvz4n"]
Jul 08 16:59:55.822304 ip-10-0-128-142 kubenswrapper[2565]: I0708 16:59:55.822271    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd5979dd-a171-4f01-adf7-da2f7ffd5a0c" path="/var/lib/kubelet/pods/cd5979dd-a171-4f01-adf7-da2f7ffd5a0c/volumes"
Jul 08 17:00:00.192844 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:00.192808    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:00:00.271783 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:00.271744    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:00:01.885535 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:01.885503    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:00:02.090285 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:02.090246    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:00:02.480229 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:02.480192    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:00:39.790667 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:39.790635    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"]
Jul 08 17:00:39.791206 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:39.791013    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd5979dd-a171-4f01-adf7-da2f7ffd5a0c" containerName="main"
Jul 08 17:00:39.791206 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:39.791024    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd5979dd-a171-4f01-adf7-da2f7ffd5a0c" containerName="main"
Jul 08 17:00:39.791206 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:39.791035    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd5979dd-a171-4f01-adf7-da2f7ffd5a0c" containerName="storage-initializer"
Jul 08 17:00:39.791206 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:39.791041    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd5979dd-a171-4f01-adf7-da2f7ffd5a0c" containerName="storage-initializer"
Jul 08 17:00:39.791206 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:39.791111    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd5979dd-a171-4f01-adf7-da2f7ffd5a0c" containerName="main"
Jul 08 17:00:39.793218 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:39.793198    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:00:39.796172 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:39.796142    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qxdr\""
Jul 08 17:00:39.796517 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:39.796501    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv8bf079eb6eda4debfb4ef5bb7817824c-kserve-self-signed-certs\""
Jul 08 17:00:39.812872 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:39.812845    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"]
Jul 08 17:00:39.822255 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:39.822225    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:00:39.861301 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:39.861280    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d18c09c-94fe-4766-9a6c-c9967962cc04-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:00:39.861411 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:39.861309    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d18c09c-94fe-4766-9a6c-c9967962cc04-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:00:39.861411 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:39.861331    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d18c09c-94fe-4766-9a6c-c9967962cc04-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:00:39.861411 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:39.861346    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d18c09c-94fe-4766-9a6c-c9967962cc04-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:00:39.861542 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:39.861443    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7whv8\" (UniqueName: \"kubernetes.io/projected/5d18c09c-94fe-4766-9a6c-c9967962cc04-kube-api-access-7whv8\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:00:39.861542 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:39.861486    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d18c09c-94fe-4766-9a6c-c9967962cc04-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:00:39.962642 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:39.962606    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7whv8\" (UniqueName: \"kubernetes.io/projected/5d18c09c-94fe-4766-9a6c-c9967962cc04-kube-api-access-7whv8\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:00:39.962787 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:39.962660    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d18c09c-94fe-4766-9a6c-c9967962cc04-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:00:39.962787 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:39.962731    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d18c09c-94fe-4766-9a6c-c9967962cc04-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:00:39.962787 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:39.962753    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d18c09c-94fe-4766-9a6c-c9967962cc04-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:00:39.962787 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:39.962772    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d18c09c-94fe-4766-9a6c-c9967962cc04-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:00:39.962787 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:39.962787    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d18c09c-94fe-4766-9a6c-c9967962cc04-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:00:39.963175 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:39.963149    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d18c09c-94fe-4766-9a6c-c9967962cc04-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:00:39.963175 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:39.963167    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d18c09c-94fe-4766-9a6c-c9967962cc04-model-cache\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:00:39.963590 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:39.963200    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d18c09c-94fe-4766-9a6c-c9967962cc04-home\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:00:39.965040 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:39.965007    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d18c09c-94fe-4766-9a6c-c9967962cc04-dshm\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:00:39.965261 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:39.965244    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d18c09c-94fe-4766-9a6c-c9967962cc04-tls-certs\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:00:39.973570 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:39.973548    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7whv8\" (UniqueName: \"kubernetes.io/projected/5d18c09c-94fe-4766-9a6c-c9967962cc04-kube-api-access-7whv8\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:00:40.263806 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:40.263770    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:00:46.625754 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:46.625718    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:00:46.635403 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:46.635374    2565 operation_generator.go:1469] "Controller attach succeeded for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:00:46.727204 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:46.727166    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:00:46.727368 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:46.727277    2565 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:00:46.731879 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:46.731858    2565 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") DevicePath \"csi-2dacefae46c7a149c270f1437d3b1a0444209eeb1f65f5a21ee1f03bbc64e3c5\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:00:46.829541 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:46.829508    2565 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/d4c2afd44bc1329e2e7018951f0ca78b5032d86e3fdf6923f6d1397e5ac2aa83/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:00:46.846345 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:46.846311    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:00:47.003368 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:47.003324    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:00:47.151902 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:47.151867    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"]
Jul 08 17:00:47.155485 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:00:47.155458    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d18c09c_94fe_4766_9a6c_c9967962cc04.slice/crio-543761c8d83f692233d93ecf49316cee08b8b14ad107af12321eaab80401fd2e WatchSource:0}: Error finding container 543761c8d83f692233d93ecf49316cee08b8b14ad107af12321eaab80401fd2e: Status 404 returned error can't find the container with id 543761c8d83f692233d93ecf49316cee08b8b14ad107af12321eaab80401fd2e
Jul 08 17:00:47.197345 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:47.197321    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4" event={"ID":"5d18c09c-94fe-4766-9a6c-c9967962cc04","Type":"ContainerStarted","Data":"543761c8d83f692233d93ecf49316cee08b8b14ad107af12321eaab80401fd2e"}
Jul 08 17:00:48.204910 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:48.204826    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4" event={"ID":"5d18c09c-94fe-4766-9a6c-c9967962cc04","Type":"ContainerStarted","Data":"02779e5a30f9330d9e798f2ea833bc24ff54ae34fa9ce8a3a72f4d02686509f5"}
Jul 08 17:00:48.240636 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:48.240582    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4" podStartSLOduration=9.240567236 podStartE2EDuration="9.240567236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:00:48.236427017 +0000 UTC m=+1584.949788466" watchObservedRunningTime="2026-07-08 17:00:48.240567236 +0000 UTC m=+1584.953928663"
Jul 08 17:00:49.004161 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:49.004130    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:00:49.052122 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:49.052085    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:00:57.003886 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:57.003848    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:00:57.005517 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:00:57.005488    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4" podUID="5d18c09c-94fe-4766-9a6c-c9967962cc04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 08 17:01:07.004686 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:01:07.004643    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4" podUID="5d18c09c-94fe-4766-9a6c-c9967962cc04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 08 17:01:17.004179 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:01:17.004131    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4" podUID="5d18c09c-94fe-4766-9a6c-c9967962cc04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 08 17:01:27.004007 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:01:27.003968    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4" podUID="5d18c09c-94fe-4766-9a6c-c9967962cc04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 08 17:01:37.004139 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:01:37.004095    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4" podUID="5d18c09c-94fe-4766-9a6c-c9967962cc04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 08 17:01:47.004807 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:01:47.004757    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4" podUID="5d18c09c-94fe-4766-9a6c-c9967962cc04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 08 17:01:57.003962 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:01:57.003918    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4" podUID="5d18c09c-94fe-4766-9a6c-c9967962cc04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 08 17:02:04.738889 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:04.738850    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:02:06.175836 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:06.175801    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:02:07.004432 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:07.004388    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4" podUID="5d18c09c-94fe-4766-9a6c-c9967962cc04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 08 17:02:17.004017 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:17.003975    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4" podUID="5d18c09c-94fe-4766-9a6c-c9967962cc04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 08 17:02:27.004022 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:27.003973    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4" podUID="5d18c09c-94fe-4766-9a6c-c9967962cc04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 08 17:02:28.252611 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:28.252576    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:02:37.013745 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:37.013712    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:02:37.021226 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:37.021204    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:02:41.418539 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:41.418492    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:02:41.431687 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:41.431644    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"]
Jul 08 17:02:41.514078 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:41.514041    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:02:53.197121 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.197044    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"]
Jul 08 17:02:53.203307 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.203284    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"
Jul 08 17:02:53.206251 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.206227    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 17:02:53.206613 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.206593    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisva690bbc929faec8bc98c767f16c003c1-kserve-self-signed-certs\""
Jul 08 17:02:53.221176 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.221144    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"]
Jul 08 17:02:53.258064 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.258027    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:02:53.265255 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.265231    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7\" (UID: \"0737e304-29dc-4bd3-9f87-c7052759ce71\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"
Jul 08 17:02:53.265359 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.265267    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7\" (UID: \"0737e304-29dc-4bd3-9f87-c7052759ce71\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"
Jul 08 17:02:53.265359 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.265297    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g6s8\" (UniqueName: \"kubernetes.io/projected/0737e304-29dc-4bd3-9f87-c7052759ce71-kube-api-access-9g6s8\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7\" (UID: \"0737e304-29dc-4bd3-9f87-c7052759ce71\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"
Jul 08 17:02:53.265430 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.265369    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7\" (UID: \"0737e304-29dc-4bd3-9f87-c7052759ce71\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"
Jul 08 17:02:53.265430 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.265415    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7\" (UID: \"0737e304-29dc-4bd3-9f87-c7052759ce71\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"
Jul 08 17:02:53.265500 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.265466    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7\" (UID: \"0737e304-29dc-4bd3-9f87-c7052759ce71\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"
Jul 08 17:02:53.265534 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.265505    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0737e304-29dc-4bd3-9f87-c7052759ce71-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7\" (UID: \"0737e304-29dc-4bd3-9f87-c7052759ce71\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"
Jul 08 17:02:53.366327 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.366291    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7\" (UID: \"0737e304-29dc-4bd3-9f87-c7052759ce71\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"
Jul 08 17:02:53.366497 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.366352    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7\" (UID: \"0737e304-29dc-4bd3-9f87-c7052759ce71\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"
Jul 08 17:02:53.366497 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.366378    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7\" (UID: \"0737e304-29dc-4bd3-9f87-c7052759ce71\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"
Jul 08 17:02:53.366497 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.366403    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0737e304-29dc-4bd3-9f87-c7052759ce71-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7\" (UID: \"0737e304-29dc-4bd3-9f87-c7052759ce71\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"
Jul 08 17:02:53.366497 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.366461    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7\" (UID: \"0737e304-29dc-4bd3-9f87-c7052759ce71\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"
Jul 08 17:02:53.366497 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.366488    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7\" (UID: \"0737e304-29dc-4bd3-9f87-c7052759ce71\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"
Jul 08 17:02:53.366855 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.366509    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g6s8\" (UniqueName: \"kubernetes.io/projected/0737e304-29dc-4bd3-9f87-c7052759ce71-kube-api-access-9g6s8\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7\" (UID: \"0737e304-29dc-4bd3-9f87-c7052759ce71\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"
Jul 08 17:02:53.366855 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.366779    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7\" (UID: \"0737e304-29dc-4bd3-9f87-c7052759ce71\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"
Jul 08 17:02:53.366855 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.366804    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-home\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7\" (UID: \"0737e304-29dc-4bd3-9f87-c7052759ce71\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"
Jul 08 17:02:53.367013 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.366848    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7\" (UID: \"0737e304-29dc-4bd3-9f87-c7052759ce71\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"
Jul 08 17:02:53.367013 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.366908    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7\" (UID: \"0737e304-29dc-4bd3-9f87-c7052759ce71\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"
Jul 08 17:02:53.368833 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.368809    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-dshm\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7\" (UID: \"0737e304-29dc-4bd3-9f87-c7052759ce71\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"
Jul 08 17:02:53.369112 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.369092    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0737e304-29dc-4bd3-9f87-c7052759ce71-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7\" (UID: \"0737e304-29dc-4bd3-9f87-c7052759ce71\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"
Jul 08 17:02:53.376769 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.376747    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g6s8\" (UniqueName: \"kubernetes.io/projected/0737e304-29dc-4bd3-9f87-c7052759ce71-kube-api-access-9g6s8\") pod \"llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7\" (UID: \"0737e304-29dc-4bd3-9f87-c7052759ce71\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"
Jul 08 17:02:53.513957 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.513867    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"
Jul 08 17:02:53.572347 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.572314    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:02:53.667588 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.667562    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"]
Jul 08 17:02:53.669936 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:02:53.669906    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0737e304_29dc_4bd3_9f87_c7052759ce71.slice/crio-0cba673b4bb91cf77b3db0c33385183d54f961498ee806dbe65212982b72fbca WatchSource:0}: Error finding container 0cba673b4bb91cf77b3db0c33385183d54f961498ee806dbe65212982b72fbca: Status 404 returned error can't find the container with id 0cba673b4bb91cf77b3db0c33385183d54f961498ee806dbe65212982b72fbca
Jul 08 17:02:53.896657 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.896564    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7" event={"ID":"0737e304-29dc-4bd3-9f87-c7052759ce71","Type":"ContainerStarted","Data":"0ce785ee3fd23b5ce4d47ee714fcc0c129a92727c61932cceb388e050009d79a"}
Jul 08 17:02:53.896657 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:53.896629    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7" event={"ID":"0737e304-29dc-4bd3-9f87-c7052759ce71","Type":"ContainerStarted","Data":"0cba673b4bb91cf77b3db0c33385183d54f961498ee806dbe65212982b72fbca"}
Jul 08 17:02:56.455425 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:56.455366    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4" podUID="5d18c09c-94fe-4766-9a6c-c9967962cc04" containerName="main" containerID="cri-o://02779e5a30f9330d9e798f2ea833bc24ff54ae34fa9ce8a3a72f4d02686509f5" gracePeriod=45
Jul 08 17:02:57.015107 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.014997    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4" podUID="5d18c09c-94fe-4766-9a6c-c9967962cc04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/health\": dial tcp 10.133.0.57:8000: connect: connection refused"
Jul 08 17:02:57.250527 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.250497    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:02:57.302193 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.302099    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d18c09c-94fe-4766-9a6c-c9967962cc04-tls-certs\") pod \"5d18c09c-94fe-4766-9a6c-c9967962cc04\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") "
Jul 08 17:02:57.302193 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.302156    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d18c09c-94fe-4766-9a6c-c9967962cc04-dshm\") pod \"5d18c09c-94fe-4766-9a6c-c9967962cc04\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") "
Jul 08 17:02:57.302193 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.302193    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7whv8\" (UniqueName: \"kubernetes.io/projected/5d18c09c-94fe-4766-9a6c-c9967962cc04-kube-api-access-7whv8\") pod \"5d18c09c-94fe-4766-9a6c-c9967962cc04\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") "
Jul 08 17:02:57.302503 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.302211    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d18c09c-94fe-4766-9a6c-c9967962cc04-model-cache\") pod \"5d18c09c-94fe-4766-9a6c-c9967962cc04\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") "
Jul 08 17:02:57.302503 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.302282    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d18c09c-94fe-4766-9a6c-c9967962cc04-home\") pod \"5d18c09c-94fe-4766-9a6c-c9967962cc04\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") "
Jul 08 17:02:57.302503 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.302390    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"5d18c09c-94fe-4766-9a6c-c9967962cc04\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") "
Jul 08 17:02:57.302503 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.302439    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d18c09c-94fe-4766-9a6c-c9967962cc04-tmp-dir\") pod \"5d18c09c-94fe-4766-9a6c-c9967962cc04\" (UID: \"5d18c09c-94fe-4766-9a6c-c9967962cc04\") "
Jul 08 17:02:57.302742 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.302506    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d18c09c-94fe-4766-9a6c-c9967962cc04-model-cache" (OuterVolumeSpecName: "model-cache") pod "5d18c09c-94fe-4766-9a6c-c9967962cc04" (UID: "5d18c09c-94fe-4766-9a6c-c9967962cc04"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:02:57.302796 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.302743    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d18c09c-94fe-4766-9a6c-c9967962cc04-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:02:57.303523 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.303491    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d18c09c-94fe-4766-9a6c-c9967962cc04-home" (OuterVolumeSpecName: "home") pod "5d18c09c-94fe-4766-9a6c-c9967962cc04" (UID: "5d18c09c-94fe-4766-9a6c-c9967962cc04"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:02:57.305262 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.305234    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d18c09c-94fe-4766-9a6c-c9967962cc04-dshm" (OuterVolumeSpecName: "dshm") pod "5d18c09c-94fe-4766-9a6c-c9967962cc04" (UID: "5d18c09c-94fe-4766-9a6c-c9967962cc04"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:02:57.305997 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.305393    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d18c09c-94fe-4766-9a6c-c9967962cc04-kube-api-access-7whv8" (OuterVolumeSpecName: "kube-api-access-7whv8") pod "5d18c09c-94fe-4766-9a6c-c9967962cc04" (UID: "5d18c09c-94fe-4766-9a6c-c9967962cc04"). InnerVolumeSpecName "kube-api-access-7whv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:02:57.305997 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.305405    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d18c09c-94fe-4766-9a6c-c9967962cc04-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5d18c09c-94fe-4766-9a6c-c9967962cc04" (UID: "5d18c09c-94fe-4766-9a6c-c9967962cc04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:02:57.307537 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.307504    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75" (OuterVolumeSpecName: "kserve-pvc-source") pod "5d18c09c-94fe-4766-9a6c-c9967962cc04" (UID: "5d18c09c-94fe-4766-9a6c-c9967962cc04"). InnerVolumeSpecName "pvc-6394c2ef-5d09-4573-b34c-dae5966e5434". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 08 17:02:57.315029 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.314998    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d18c09c-94fe-4766-9a6c-c9967962cc04-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5d18c09c-94fe-4766-9a6c-c9967962cc04" (UID: "5d18c09c-94fe-4766-9a6c-c9967962cc04"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:02:57.404017 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.403958    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d18c09c-94fe-4766-9a6c-c9967962cc04-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:02:57.404017 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.404003    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7whv8\" (UniqueName: \"kubernetes.io/projected/5d18c09c-94fe-4766-9a6c-c9967962cc04-kube-api-access-7whv8\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:02:57.404017 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.404022    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d18c09c-94fe-4766-9a6c-c9967962cc04-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:02:57.404328 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.404068    2565 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") on node \"ip-10-0-128-142.ec2.internal\" "
Jul 08 17:02:57.404328 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.404083    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d18c09c-94fe-4766-9a6c-c9967962cc04-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:02:57.404328 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.404099    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d18c09c-94fe-4766-9a6c-c9967962cc04-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:02:57.437649 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.437627    2565 operation_generator.go:895] UnmountDevice succeeded for volume "pvc-6394c2ef-5d09-4573-b34c-dae5966e5434" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75") on node "ip-10-0-128-142.ec2.internal"
Jul 08 17:02:57.505090 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.505057    2565 reconciler_common.go:299] "Volume detached for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"csi-2dacefae46c7a149c270f1437d3b1a0444209eeb1f65f5a21ee1f03bbc64e3c5\""
Jul 08 17:02:57.919377 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.919337    2565 generic.go:356] "Generic (PLEG): container finished" podID="5d18c09c-94fe-4766-9a6c-c9967962cc04" containerID="02779e5a30f9330d9e798f2ea833bc24ff54ae34fa9ce8a3a72f4d02686509f5" exitCode=0
Jul 08 17:02:57.919569 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.919383    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"
Jul 08 17:02:57.919569 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.919389    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4" event={"ID":"5d18c09c-94fe-4766-9a6c-c9967962cc04","Type":"ContainerDied","Data":"02779e5a30f9330d9e798f2ea833bc24ff54ae34fa9ce8a3a72f4d02686509f5"}
Jul 08 17:02:57.919569 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.919443    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4" event={"ID":"5d18c09c-94fe-4766-9a6c-c9967962cc04","Type":"ContainerDied","Data":"543761c8d83f692233d93ecf49316cee08b8b14ad107af12321eaab80401fd2e"}
Jul 08 17:02:57.919569 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.919471    2565 scope.go:117] "RemoveContainer" containerID="02779e5a30f9330d9e798f2ea833bc24ff54ae34fa9ce8a3a72f4d02686509f5"
Jul 08 17:02:57.928497 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.928479    2565 scope.go:117] "RemoveContainer" containerID="02779e5a30f9330d9e798f2ea833bc24ff54ae34fa9ce8a3a72f4d02686509f5"
Jul 08 17:02:57.928811 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:02:57.928792    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02779e5a30f9330d9e798f2ea833bc24ff54ae34fa9ce8a3a72f4d02686509f5\": container with ID starting with 02779e5a30f9330d9e798f2ea833bc24ff54ae34fa9ce8a3a72f4d02686509f5 not found: ID does not exist" containerID="02779e5a30f9330d9e798f2ea833bc24ff54ae34fa9ce8a3a72f4d02686509f5"
Jul 08 17:02:57.928912 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.928818    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02779e5a30f9330d9e798f2ea833bc24ff54ae34fa9ce8a3a72f4d02686509f5"} err="failed to get container status \"02779e5a30f9330d9e798f2ea833bc24ff54ae34fa9ce8a3a72f4d02686509f5\": rpc error: code = NotFound desc = could not find container \"02779e5a30f9330d9e798f2ea833bc24ff54ae34fa9ce8a3a72f4d02686509f5\": container with ID starting with 02779e5a30f9330d9e798f2ea833bc24ff54ae34fa9ce8a3a72f4d02686509f5 not found: ID does not exist"
Jul 08 17:02:57.946274 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.946242    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"]
Jul 08 17:02:57.952362 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:57.952331    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-59b9d263-kserve-585587bc9df8wz4"]
Jul 08 17:02:58.926947 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:58.926914    2565 generic.go:356] "Generic (PLEG): container finished" podID="0737e304-29dc-4bd3-9f87-c7052759ce71" containerID="0ce785ee3fd23b5ce4d47ee714fcc0c129a92727c61932cceb388e050009d79a" exitCode=0
Jul 08 17:02:58.927413 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:58.926967    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7" event={"ID":"0737e304-29dc-4bd3-9f87-c7052759ce71","Type":"ContainerDied","Data":"0ce785ee3fd23b5ce4d47ee714fcc0c129a92727c61932cceb388e050009d79a"}
Jul 08 17:02:59.821527 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:59.821490    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d18c09c-94fe-4766-9a6c-c9967962cc04" path="/var/lib/kubelet/pods/5d18c09c-94fe-4766-9a6c-c9967962cc04/volumes"
Jul 08 17:02:59.934755 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:59.934723    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7" event={"ID":"0737e304-29dc-4bd3-9f87-c7052759ce71","Type":"ContainerStarted","Data":"59815841ac11e27f65e32a3f55be7e6fd18d9313e6a0ad83afb4734ac3c306c6"}
Jul 08 17:02:59.969797 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:02:59.969747    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7" podStartSLOduration=2.448412937 podStartE2EDuration="6.969731714s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.521318777s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:02:59.966312843 +0000 UTC m=+1716.679674271" watchObservedRunningTime="2026-07-08 17:02:59.969731714 +0000 UTC m=+1716.683093141"
Jul 08 17:03:00.514634 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:00.514596    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"
Jul 08 17:03:01.945067 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:01.945029    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:03:03.514714 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:03.514660    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"
Jul 08 17:03:03.516482 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:03.516451    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7" podUID="0737e304-29dc-4bd3-9f87-c7052759ce71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 08 17:03:10.153694 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.152876    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"]
Jul 08 17:03:10.153694 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.153545    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d18c09c-94fe-4766-9a6c-c9967962cc04" containerName="main"
Jul 08 17:03:10.153694 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.153563    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d18c09c-94fe-4766-9a6c-c9967962cc04" containerName="main"
Jul 08 17:03:10.154392 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.153702    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d18c09c-94fe-4766-9a6c-c9967962cc04" containerName="main"
Jul 08 17:03:10.160265 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.160239    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"]
Jul 08 17:03:10.160513 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.160338    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:10.164295 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.164270    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-e8706282-kserve-dockercfg-7bqrf\""
Jul 08 17:03:10.164833 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.164369    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvb19f98874e050eec8ca94d49676113f0-kserve-self-signed-certs\""
Jul 08 17:03:10.172469 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.172450    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"]
Jul 08 17:03:10.172643 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.172628    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"]
Jul 08 17:03:10.172827 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.172640    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:03:10.173783 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.173753    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:03:10.225426 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.225380    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e727c0f5-7c04-44bc-9204-a826f79d6587-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:03:10.225593 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.225451    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dfjz\" (UniqueName: \"kubernetes.io/projected/e727c0f5-7c04-44bc-9204-a826f79d6587-kube-api-access-2dfjz\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:03:10.225593 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.225501    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mgjw\" (UniqueName: \"kubernetes.io/projected/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-kube-api-access-7mgjw\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:10.225593 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.225538    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:10.225593 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.225564    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e727c0f5-7c04-44bc-9204-a826f79d6587-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:03:10.225593 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.225581    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e727c0f5-7c04-44bc-9204-a826f79d6587-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:03:10.225896 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.225729    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e727c0f5-7c04-44bc-9204-a826f79d6587-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:03:10.225896 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.225773    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:10.225896 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.225798    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:10.225896 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.225822    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:10.225896 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.225872    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:10.226140 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.225895    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e727c0f5-7c04-44bc-9204-a826f79d6587-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:03:10.326477 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.326437    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e727c0f5-7c04-44bc-9204-a826f79d6587-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:03:10.326477 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.326481    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:10.326744 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.326503    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:10.326744 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.326527    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:10.326744 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.326571    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:10.326744 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.326595    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e727c0f5-7c04-44bc-9204-a826f79d6587-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:03:10.326744 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.326621    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e727c0f5-7c04-44bc-9204-a826f79d6587-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:03:10.326744 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.326670    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dfjz\" (UniqueName: \"kubernetes.io/projected/e727c0f5-7c04-44bc-9204-a826f79d6587-kube-api-access-2dfjz\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:03:10.327068 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.326753    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mgjw\" (UniqueName: \"kubernetes.io/projected/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-kube-api-access-7mgjw\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:10.327068 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.326783    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:10.327068 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.326813    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e727c0f5-7c04-44bc-9204-a826f79d6587-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:03:10.327068 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.326835    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e727c0f5-7c04-44bc-9204-a826f79d6587-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:03:10.327068 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.326936    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e727c0f5-7c04-44bc-9204-a826f79d6587-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:03:10.327068 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.326936    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:10.327068 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.326978    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-model-cache\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:10.327450 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.327164    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e727c0f5-7c04-44bc-9204-a826f79d6587-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:03:10.327450 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.327214    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:10.327450 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.327435    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e727c0f5-7c04-44bc-9204-a826f79d6587-home\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:03:10.329354 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.329324    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e727c0f5-7c04-44bc-9204-a826f79d6587-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:03:10.329529 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.329510    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-tls-certs\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:10.329591 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.329523    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e727c0f5-7c04-44bc-9204-a826f79d6587-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:03:10.329591 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.329513    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-dshm\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:10.337365 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.337322    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dfjz\" (UniqueName: \"kubernetes.io/projected/e727c0f5-7c04-44bc-9204-a826f79d6587-kube-api-access-2dfjz\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:03:10.338034 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.338006    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mgjw\" (UniqueName: \"kubernetes.io/projected/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-kube-api-access-7mgjw\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:10.599943 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:10.599903    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:03:13.515151 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:13.515103    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7" podUID="0737e304-29dc-4bd3-9f87-c7052759ce71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 08 17:03:19.821575 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:19.821537    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:19.830204 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:19.830177    2565 operation_generator.go:1469] "Controller attach succeeded for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:19.923144 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:19.923109    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:03:19.923346 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:19.923248    2565 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:03:19.928001 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:19.927963    2565 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") DevicePath \"csi-2dacefae46c7a149c270f1437d3b1a0444209eeb1f65f5a21ee1f03bbc64e3c5\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:03:20.019351 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:20.019301    2565 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/d4c2afd44bc1329e2e7018951f0ca78b5032d86e3fdf6923f6d1397e5ac2aa83/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:03:20.038834 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:20.038806    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:03:20.090172 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:20.090083    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:03:20.124656 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:20.124630    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:20.124832 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:20.124786    2565 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") DevicePath \"csi-2dacefae46c7a149c270f1437d3b1a0444209eeb1f65f5a21ee1f03bbc64e3c5\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:20.130965 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:20.130943    2565 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") DevicePath \"csi-2dacefae46c7a149c270f1437d3b1a0444209eeb1f65f5a21ee1f03bbc64e3c5\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:20.147849 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:20.147640    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:20.265082 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:20.265048    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"]
Jul 08 17:03:20.265993 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:03:20.265962    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode727c0f5_7c04_44bc_9204_a826f79d6587.slice/crio-e7a25967c25c081b051905be4e64bed76317a600d63efa19d1341951781ed92e WatchSource:0}: Error finding container e7a25967c25c081b051905be4e64bed76317a600d63efa19d1341951781ed92e: Status 404 returned error can't find the container with id e7a25967c25c081b051905be4e64bed76317a600d63efa19d1341951781ed92e
Jul 08 17:03:20.382487 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:20.382396    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:20.533854 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:20.533825    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"]
Jul 08 17:03:20.535757 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:03:20.535726    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb294d5b0_37b6_428a_b05c_dee4cf6d4e75.slice/crio-022894c8d472aeb975c573bfc0a217aff825dc4b9bcc5849edc9e5379f1aedd0 WatchSource:0}: Error finding container 022894c8d472aeb975c573bfc0a217aff825dc4b9bcc5849edc9e5379f1aedd0: Status 404 returned error can't find the container with id 022894c8d472aeb975c573bfc0a217aff825dc4b9bcc5849edc9e5379f1aedd0
Jul 08 17:03:20.538063 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:20.538036    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 17:03:21.076693 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:21.076583    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k" event={"ID":"b294d5b0-37b6-428a-b05c-dee4cf6d4e75","Type":"ContainerStarted","Data":"022894c8d472aeb975c573bfc0a217aff825dc4b9bcc5849edc9e5379f1aedd0"}
Jul 08 17:03:21.078630 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:21.078533    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv" event={"ID":"e727c0f5-7c04-44bc-9204-a826f79d6587","Type":"ContainerStarted","Data":"7085a4378028acf88c7f6dc1408b50b3149331ec7c73c49556e99ad6e62f5b93"}
Jul 08 17:03:21.078630 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:21.078574    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv" event={"ID":"e727c0f5-7c04-44bc-9204-a826f79d6587","Type":"ContainerStarted","Data":"e7a25967c25c081b051905be4e64bed76317a600d63efa19d1341951781ed92e"}
Jul 08 17:03:21.105508 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:21.105450    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv" podStartSLOduration=11.105429849 podStartE2EDuration="11.105429849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:03:21.101552599 +0000 UTC m=+1737.814914030" watchObservedRunningTime="2026-07-08 17:03:21.105429849 +0000 UTC m=+1737.818791281"
Jul 08 17:03:22.087106 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:22.087076    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k" event={"ID":"b294d5b0-37b6-428a-b05c-dee4cf6d4e75","Type":"ContainerStarted","Data":"58d1c4cbf5d3e3b5b981fca145d486a9bb0bb9b3c53b28f4d36d52d018f25ef8"}
Jul 08 17:03:22.087523 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:22.087367    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:23.096893 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:23.096855    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k" event={"ID":"b294d5b0-37b6-428a-b05c-dee4cf6d4e75","Type":"ContainerStarted","Data":"1fd716e714e7d51155e5580241076f6e30785d6747dbba638ffe7302b629109c"}
Jul 08 17:03:23.127197 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:23.127146    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k" podStartSLOduration=12.344556053 podStartE2EDuration="13.127125517s" totalImagesPullingTime="782.569464ms" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:03:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:03:23.122449174 +0000 UTC m=+1739.835810625" watchObservedRunningTime="2026-07-08 17:03:23.127125517 +0000 UTC m=+1739.840486945"
Jul 08 17:03:23.515125 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:23.515072    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7" podUID="0737e304-29dc-4bd3-9f87-c7052759ce71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 08 17:03:27.465868 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:27.465830    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:03:30.090918 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:30.090875    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:03:30.091404 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:30.090984    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:03:30.092824 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:30.092788    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv" podUID="e727c0f5-7c04-44bc-9204-a826f79d6587" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 08 17:03:30.382952 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:30.382845    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:30.383112 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:30.382962    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:30.384349 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:30.384320    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k" podUID="b294d5b0-37b6-428a-b05c-dee4cf6d4e75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Jul 08 17:03:33.514632 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:33.514582    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7" podUID="0737e304-29dc-4bd3-9f87-c7052759ce71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 08 17:03:40.090910 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:40.090850    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv" podUID="e727c0f5-7c04-44bc-9204-a826f79d6587" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 08 17:03:40.384057 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:40.383920    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k" podUID="b294d5b0-37b6-428a-b05c-dee4cf6d4e75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Jul 08 17:03:40.397156 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:40.397123    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:03:43.514531 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:43.514483    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7" podUID="0737e304-29dc-4bd3-9f87-c7052759ce71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 08 17:03:50.090946 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:50.090904    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv" podUID="e727c0f5-7c04-44bc-9204-a826f79d6587" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 08 17:03:50.383088 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:50.382971    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k" podUID="b294d5b0-37b6-428a-b05c-dee4cf6d4e75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Jul 08 17:03:53.515485 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:03:53.515437    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7" podUID="0737e304-29dc-4bd3-9f87-c7052759ce71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 08 17:04:00.091537 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:04:00.091483    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv" podUID="e727c0f5-7c04-44bc-9204-a826f79d6587" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 08 17:04:00.383952 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:04:00.383849    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k" podUID="b294d5b0-37b6-428a-b05c-dee4cf6d4e75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Jul 08 17:04:03.515331 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:04:03.515273    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7" podUID="0737e304-29dc-4bd3-9f87-c7052759ce71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 08 17:04:10.090883 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:04:10.090833    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv" podUID="e727c0f5-7c04-44bc-9204-a826f79d6587" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 08 17:04:10.383135 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:04:10.383026    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k" podUID="b294d5b0-37b6-428a-b05c-dee4cf6d4e75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Jul 08 17:04:13.514538 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:04:13.514489    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7" podUID="0737e304-29dc-4bd3-9f87-c7052759ce71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 08 17:04:20.091097 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:04:20.091039    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv" podUID="e727c0f5-7c04-44bc-9204-a826f79d6587" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 08 17:04:20.383320 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:04:20.383203    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k" podUID="b294d5b0-37b6-428a-b05c-dee4cf6d4e75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Jul 08 17:04:23.515381 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:04:23.515333    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7" podUID="0737e304-29dc-4bd3-9f87-c7052759ce71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 08 17:04:30.090575 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:04:30.090526    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv" podUID="e727c0f5-7c04-44bc-9204-a826f79d6587" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 08 17:04:30.382982 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:04:30.382862    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k" podUID="b294d5b0-37b6-428a-b05c-dee4cf6d4e75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Jul 08 17:04:33.514668 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:04:33.514612    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7" podUID="0737e304-29dc-4bd3-9f87-c7052759ce71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 08 17:04:40.091098 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:04:40.091053    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv" podUID="e727c0f5-7c04-44bc-9204-a826f79d6587" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 08 17:04:40.383915 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:04:40.383804    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k" podUID="b294d5b0-37b6-428a-b05c-dee4cf6d4e75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Jul 08 17:04:43.514598 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:04:43.514550    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7" podUID="0737e304-29dc-4bd3-9f87-c7052759ce71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 08 17:04:50.091577 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:04:50.091530    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv" podUID="e727c0f5-7c04-44bc-9204-a826f79d6587" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 08 17:04:50.383133 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:04:50.383023    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k" podUID="b294d5b0-37b6-428a-b05c-dee4cf6d4e75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Jul 08 17:04:53.514573 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:04:53.514520    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7" podUID="0737e304-29dc-4bd3-9f87-c7052759ce71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 08 17:05:00.090688 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:05:00.090622    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv" podUID="e727c0f5-7c04-44bc-9204-a826f79d6587" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 08 17:05:00.383893 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:05:00.383765    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k" podUID="b294d5b0-37b6-428a-b05c-dee4cf6d4e75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Jul 08 17:05:01.370660 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:05:01.370618    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:05:03.515070 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:05:03.515027    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7" podUID="0737e304-29dc-4bd3-9f87-c7052759ce71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 08 17:05:10.090972 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:05:10.090917    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv" podUID="e727c0f5-7c04-44bc-9204-a826f79d6587" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 08 17:05:10.383594 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:05:10.383486    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k" podUID="b294d5b0-37b6-428a-b05c-dee4cf6d4e75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Jul 08 17:05:13.514924 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:05:13.514869    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7" podUID="0737e304-29dc-4bd3-9f87-c7052759ce71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 08 17:05:20.091106 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:05:20.091052    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv" podUID="e727c0f5-7c04-44bc-9204-a826f79d6587" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 08 17:05:20.383656 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:05:20.383544    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k" podUID="b294d5b0-37b6-428a-b05c-dee4cf6d4e75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Jul 08 17:05:23.515179 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:05:23.515128    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7" podUID="0737e304-29dc-4bd3-9f87-c7052759ce71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 08 17:05:30.090755 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:05:30.090702    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv" podUID="e727c0f5-7c04-44bc-9204-a826f79d6587" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 08 17:05:30.382991 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:05:30.382889    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k" podUID="b294d5b0-37b6-428a-b05c-dee4cf6d4e75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Jul 08 17:05:33.524587 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:05:33.524549    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"
Jul 08 17:05:33.532409 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:05:33.532387    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"
Jul 08 17:05:40.090629 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:05:40.090586    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv" podUID="e727c0f5-7c04-44bc-9204-a826f79d6587" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 08 17:05:40.383200 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:05:40.383083    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k" podUID="b294d5b0-37b6-428a-b05c-dee4cf6d4e75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Jul 08 17:05:44.762055 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:05:44.761973    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:05:44.763420 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:05:44.763384    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"]
Jul 08 17:05:44.864446 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:05:44.864409    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:05:50.091141 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:05:50.091102    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv" podUID="e727c0f5-7c04-44bc-9204-a826f79d6587" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 08 17:05:50.383092 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:05:50.382983    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k" podUID="b294d5b0-37b6-428a-b05c-dee4cf6d4e75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Jul 08 17:05:53.366542 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:05:53.366510    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:05:59.800090 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:05:59.800047    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7" podUID="0737e304-29dc-4bd3-9f87-c7052759ce71" containerName="main" containerID="cri-o://59815841ac11e27f65e32a3f55be7e6fd18d9313e6a0ad83afb4734ac3c306c6" gracePeriod=45
Jul 08 17:06:00.090630 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:00.090545    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv" podUID="e727c0f5-7c04-44bc-9204-a826f79d6587" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 08 17:06:00.383482 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:00.383368    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k" podUID="b294d5b0-37b6-428a-b05c-dee4cf6d4e75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Jul 08 17:06:00.525611 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:00.525571    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7" podUID="0737e304-29dc-4bd3-9f87-c7052759ce71" containerName="main" probeResult="failure" output="Get \"https://10.133.0.58:8000/health\": dial tcp 10.133.0.58:8000: connect: connection refused"
Jul 08 17:06:01.074861 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.074835    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"
Jul 08 17:06:01.140999 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.140911    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-model-cache\") pod \"0737e304-29dc-4bd3-9f87-c7052759ce71\" (UID: \"0737e304-29dc-4bd3-9f87-c7052759ce71\") "
Jul 08 17:06:01.140999 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.140950    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0737e304-29dc-4bd3-9f87-c7052759ce71-tls-certs\") pod \"0737e304-29dc-4bd3-9f87-c7052759ce71\" (UID: \"0737e304-29dc-4bd3-9f87-c7052759ce71\") "
Jul 08 17:06:01.140999 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.140980    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-dshm\") pod \"0737e304-29dc-4bd3-9f87-c7052759ce71\" (UID: \"0737e304-29dc-4bd3-9f87-c7052759ce71\") "
Jul 08 17:06:01.140999 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.140997    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9g6s8\" (UniqueName: \"kubernetes.io/projected/0737e304-29dc-4bd3-9f87-c7052759ce71-kube-api-access-9g6s8\") pod \"0737e304-29dc-4bd3-9f87-c7052759ce71\" (UID: \"0737e304-29dc-4bd3-9f87-c7052759ce71\") "
Jul 08 17:06:01.141327 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.141038    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-kserve-provision-location\") pod \"0737e304-29dc-4bd3-9f87-c7052759ce71\" (UID: \"0737e304-29dc-4bd3-9f87-c7052759ce71\") "
Jul 08 17:06:01.141327 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.141063    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-tmp-dir\") pod \"0737e304-29dc-4bd3-9f87-c7052759ce71\" (UID: \"0737e304-29dc-4bd3-9f87-c7052759ce71\") "
Jul 08 17:06:01.141327 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.141100    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-home\") pod \"0737e304-29dc-4bd3-9f87-c7052759ce71\" (UID: \"0737e304-29dc-4bd3-9f87-c7052759ce71\") "
Jul 08 17:06:01.141327 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.141175    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-model-cache" (OuterVolumeSpecName: "model-cache") pod "0737e304-29dc-4bd3-9f87-c7052759ce71" (UID: "0737e304-29dc-4bd3-9f87-c7052759ce71"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:06:01.141536 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.141437    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:06:01.142117 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.142079    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-home" (OuterVolumeSpecName: "home") pod "0737e304-29dc-4bd3-9f87-c7052759ce71" (UID: "0737e304-29dc-4bd3-9f87-c7052759ce71"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:06:01.143187 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.143160    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0737e304-29dc-4bd3-9f87-c7052759ce71-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0737e304-29dc-4bd3-9f87-c7052759ce71" (UID: "0737e304-29dc-4bd3-9f87-c7052759ce71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:06:01.143404 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.143383    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0737e304-29dc-4bd3-9f87-c7052759ce71-kube-api-access-9g6s8" (OuterVolumeSpecName: "kube-api-access-9g6s8") pod "0737e304-29dc-4bd3-9f87-c7052759ce71" (UID: "0737e304-29dc-4bd3-9f87-c7052759ce71"). InnerVolumeSpecName "kube-api-access-9g6s8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:06:01.144236 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.144209    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-dshm" (OuterVolumeSpecName: "dshm") pod "0737e304-29dc-4bd3-9f87-c7052759ce71" (UID: "0737e304-29dc-4bd3-9f87-c7052759ce71"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:06:01.159494 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.159462    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0737e304-29dc-4bd3-9f87-c7052759ce71" (UID: "0737e304-29dc-4bd3-9f87-c7052759ce71"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:06:01.162430 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.162402    2565 generic.go:356] "Generic (PLEG): container finished" podID="0737e304-29dc-4bd3-9f87-c7052759ce71" containerID="59815841ac11e27f65e32a3f55be7e6fd18d9313e6a0ad83afb4734ac3c306c6" exitCode=0
Jul 08 17:06:01.162520 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.162463    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"
Jul 08 17:06:01.162520 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.162479    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7" event={"ID":"0737e304-29dc-4bd3-9f87-c7052759ce71","Type":"ContainerDied","Data":"59815841ac11e27f65e32a3f55be7e6fd18d9313e6a0ad83afb4734ac3c306c6"}
Jul 08 17:06:01.162604 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.162535    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7" event={"ID":"0737e304-29dc-4bd3-9f87-c7052759ce71","Type":"ContainerDied","Data":"0cba673b4bb91cf77b3db0c33385183d54f961498ee806dbe65212982b72fbca"}
Jul 08 17:06:01.162604 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.162560    2565 scope.go:117] "RemoveContainer" containerID="59815841ac11e27f65e32a3f55be7e6fd18d9313e6a0ad83afb4734ac3c306c6"
Jul 08 17:06:01.171538 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.171521    2565 scope.go:117] "RemoveContainer" containerID="0ce785ee3fd23b5ce4d47ee714fcc0c129a92727c61932cceb388e050009d79a"
Jul 08 17:06:01.180069 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.180053    2565 scope.go:117] "RemoveContainer" containerID="59815841ac11e27f65e32a3f55be7e6fd18d9313e6a0ad83afb4734ac3c306c6"
Jul 08 17:06:01.180319 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:06:01.180302    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59815841ac11e27f65e32a3f55be7e6fd18d9313e6a0ad83afb4734ac3c306c6\": container with ID starting with 59815841ac11e27f65e32a3f55be7e6fd18d9313e6a0ad83afb4734ac3c306c6 not found: ID does not exist" containerID="59815841ac11e27f65e32a3f55be7e6fd18d9313e6a0ad83afb4734ac3c306c6"
Jul 08 17:06:01.180364 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.180327    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59815841ac11e27f65e32a3f55be7e6fd18d9313e6a0ad83afb4734ac3c306c6"} err="failed to get container status \"59815841ac11e27f65e32a3f55be7e6fd18d9313e6a0ad83afb4734ac3c306c6\": rpc error: code = NotFound desc = could not find container \"59815841ac11e27f65e32a3f55be7e6fd18d9313e6a0ad83afb4734ac3c306c6\": container with ID starting with 59815841ac11e27f65e32a3f55be7e6fd18d9313e6a0ad83afb4734ac3c306c6 not found: ID does not exist"
Jul 08 17:06:01.180364 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.180345    2565 scope.go:117] "RemoveContainer" containerID="0ce785ee3fd23b5ce4d47ee714fcc0c129a92727c61932cceb388e050009d79a"
Jul 08 17:06:01.180598 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:06:01.180582    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ce785ee3fd23b5ce4d47ee714fcc0c129a92727c61932cceb388e050009d79a\": container with ID starting with 0ce785ee3fd23b5ce4d47ee714fcc0c129a92727c61932cceb388e050009d79a not found: ID does not exist" containerID="0ce785ee3fd23b5ce4d47ee714fcc0c129a92727c61932cceb388e050009d79a"
Jul 08 17:06:01.180652 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.180604    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ce785ee3fd23b5ce4d47ee714fcc0c129a92727c61932cceb388e050009d79a"} err="failed to get container status \"0ce785ee3fd23b5ce4d47ee714fcc0c129a92727c61932cceb388e050009d79a\": rpc error: code = NotFound desc = could not find container \"0ce785ee3fd23b5ce4d47ee714fcc0c129a92727c61932cceb388e050009d79a\": container with ID starting with 0ce785ee3fd23b5ce4d47ee714fcc0c129a92727c61932cceb388e050009d79a not found: ID does not exist"
Jul 08 17:06:01.210226 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.210203    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0737e304-29dc-4bd3-9f87-c7052759ce71" (UID: "0737e304-29dc-4bd3-9f87-c7052759ce71"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:06:01.242482 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.242458    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0737e304-29dc-4bd3-9f87-c7052759ce71-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:06:01.242482 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.242480    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:06:01.242599 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.242489    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g6s8\" (UniqueName: \"kubernetes.io/projected/0737e304-29dc-4bd3-9f87-c7052759ce71-kube-api-access-9g6s8\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:06:01.242599 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.242498    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-kserve-provision-location\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:06:01.242599 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.242507    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:06:01.242599 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.242516    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0737e304-29dc-4bd3-9f87-c7052759ce71-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:06:01.497506 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.497464    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"]
Jul 08 17:06:01.509381 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.509349    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-87882a8e-kserve-66b95f57dfnrhp7"]
Jul 08 17:06:01.822026 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:01.821931    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0737e304-29dc-4bd3-9f87-c7052759ce71" path="/var/lib/kubelet/pods/0737e304-29dc-4bd3-9f87-c7052759ce71/volumes"
Jul 08 17:06:10.090888 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:10.090829    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv" podUID="e727c0f5-7c04-44bc-9204-a826f79d6587" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 08 17:06:10.383951 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:10.383831    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k" podUID="b294d5b0-37b6-428a-b05c-dee4cf6d4e75" containerName="main" probeResult="failure" output="Get \"https://10.133.0.59:8001/health\": dial tcp 10.133.0.59:8001: connect: connection refused"
Jul 08 17:06:16.000207 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.000173    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"]
Jul 08 17:06:16.000571 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.000556    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0737e304-29dc-4bd3-9f87-c7052759ce71" containerName="storage-initializer"
Jul 08 17:06:16.000571 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.000567    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0737e304-29dc-4bd3-9f87-c7052759ce71" containerName="storage-initializer"
Jul 08 17:06:16.000648 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.000584    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0737e304-29dc-4bd3-9f87-c7052759ce71" containerName="main"
Jul 08 17:06:16.000648 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.000590    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0737e304-29dc-4bd3-9f87-c7052759ce71" containerName="main"
Jul 08 17:06:16.000729 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.000654    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0737e304-29dc-4bd3-9f87-c7052759ce71" containerName="main"
Jul 08 17:06:16.004222 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.004203    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:06:16.008273 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.008252    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-test-kserve-self-signed-certs\""
Jul 08 17:06:16.008391 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.008362    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 17:06:16.018489 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.018463    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"]
Jul 08 17:06:16.026034 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.026011    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:06:16.099088 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.099038    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-86c9c75755-7p6kx\" (UID: \"2d1929ff-8377-47e1-93cd-6d955cadc959\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:06:16.099088 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.099097    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdwm7\" (UniqueName: \"kubernetes.io/projected/2d1929ff-8377-47e1-93cd-6d955cadc959-kube-api-access-xdwm7\") pod \"custom-route-timeout-test-kserve-86c9c75755-7p6kx\" (UID: \"2d1929ff-8377-47e1-93cd-6d955cadc959\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:06:16.099339 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.099201    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-model-cache\") pod \"custom-route-timeout-test-kserve-86c9c75755-7p6kx\" (UID: \"2d1929ff-8377-47e1-93cd-6d955cadc959\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:06:16.099339 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.099260    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-tmp-dir\") pod \"custom-route-timeout-test-kserve-86c9c75755-7p6kx\" (UID: \"2d1929ff-8377-47e1-93cd-6d955cadc959\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:06:16.099339 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.099288    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-dshm\") pod \"custom-route-timeout-test-kserve-86c9c75755-7p6kx\" (UID: \"2d1929ff-8377-47e1-93cd-6d955cadc959\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:06:16.099339 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.099315    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-home\") pod \"custom-route-timeout-test-kserve-86c9c75755-7p6kx\" (UID: \"2d1929ff-8377-47e1-93cd-6d955cadc959\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:06:16.099339 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.099330    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1929ff-8377-47e1-93cd-6d955cadc959-tls-certs\") pod \"custom-route-timeout-test-kserve-86c9c75755-7p6kx\" (UID: \"2d1929ff-8377-47e1-93cd-6d955cadc959\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:06:16.200391 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.200349    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-home\") pod \"custom-route-timeout-test-kserve-86c9c75755-7p6kx\" (UID: \"2d1929ff-8377-47e1-93cd-6d955cadc959\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:06:16.200391 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.200395    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1929ff-8377-47e1-93cd-6d955cadc959-tls-certs\") pod \"custom-route-timeout-test-kserve-86c9c75755-7p6kx\" (UID: \"2d1929ff-8377-47e1-93cd-6d955cadc959\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:06:16.200626 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.200440    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-86c9c75755-7p6kx\" (UID: \"2d1929ff-8377-47e1-93cd-6d955cadc959\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:06:16.200626 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.200480    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdwm7\" (UniqueName: \"kubernetes.io/projected/2d1929ff-8377-47e1-93cd-6d955cadc959-kube-api-access-xdwm7\") pod \"custom-route-timeout-test-kserve-86c9c75755-7p6kx\" (UID: \"2d1929ff-8377-47e1-93cd-6d955cadc959\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:06:16.200775 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.200712    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-model-cache\") pod \"custom-route-timeout-test-kserve-86c9c75755-7p6kx\" (UID: \"2d1929ff-8377-47e1-93cd-6d955cadc959\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:06:16.200842 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.200825    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-tmp-dir\") pod \"custom-route-timeout-test-kserve-86c9c75755-7p6kx\" (UID: \"2d1929ff-8377-47e1-93cd-6d955cadc959\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:06:16.200902 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.200840    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-home\") pod \"custom-route-timeout-test-kserve-86c9c75755-7p6kx\" (UID: \"2d1929ff-8377-47e1-93cd-6d955cadc959\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:06:16.200902 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.200873    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-dshm\") pod \"custom-route-timeout-test-kserve-86c9c75755-7p6kx\" (UID: \"2d1929ff-8377-47e1-93cd-6d955cadc959\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:06:16.200991 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.200912    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-kserve-provision-location\") pod \"custom-route-timeout-test-kserve-86c9c75755-7p6kx\" (UID: \"2d1929ff-8377-47e1-93cd-6d955cadc959\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:06:16.201027 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.201005    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-model-cache\") pod \"custom-route-timeout-test-kserve-86c9c75755-7p6kx\" (UID: \"2d1929ff-8377-47e1-93cd-6d955cadc959\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:06:16.201098 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.201076    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-tmp-dir\") pod \"custom-route-timeout-test-kserve-86c9c75755-7p6kx\" (UID: \"2d1929ff-8377-47e1-93cd-6d955cadc959\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:06:16.203036 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.203003    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-dshm\") pod \"custom-route-timeout-test-kserve-86c9c75755-7p6kx\" (UID: \"2d1929ff-8377-47e1-93cd-6d955cadc959\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:06:16.203132 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.203117    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1929ff-8377-47e1-93cd-6d955cadc959-tls-certs\") pod \"custom-route-timeout-test-kserve-86c9c75755-7p6kx\" (UID: \"2d1929ff-8377-47e1-93cd-6d955cadc959\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:06:16.211142 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.211118    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdwm7\" (UniqueName: \"kubernetes.io/projected/2d1929ff-8377-47e1-93cd-6d955cadc959-kube-api-access-xdwm7\") pod \"custom-route-timeout-test-kserve-86c9c75755-7p6kx\" (UID: \"2d1929ff-8377-47e1-93cd-6d955cadc959\") " pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:06:16.314417 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.314310    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:06:16.462557 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.462521    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:06:16.462956 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:16.462932    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"]
Jul 08 17:06:16.463983 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:06:16.463948    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1929ff_8377_47e1_93cd_6d955cadc959.slice/crio-33d206715d2eb3e908a99492ab12fdf77773490a6260ec53066dea3c627c0dad WatchSource:0}: Error finding container 33d206715d2eb3e908a99492ab12fdf77773490a6260ec53066dea3c627c0dad: Status 404 returned error can't find the container with id 33d206715d2eb3e908a99492ab12fdf77773490a6260ec53066dea3c627c0dad
Jul 08 17:06:17.255234 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:17.255191    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx" event={"ID":"2d1929ff-8377-47e1-93cd-6d955cadc959","Type":"ContainerStarted","Data":"1786a3f4117071cdca2bd596dd6a23ce0db619a72b1108870aca3131adc42499"}
Jul 08 17:06:17.255614 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:17.255291    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx" event={"ID":"2d1929ff-8377-47e1-93cd-6d955cadc959","Type":"ContainerStarted","Data":"33d206715d2eb3e908a99492ab12fdf77773490a6260ec53066dea3c627c0dad"}
Jul 08 17:06:20.100997 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:20.100965    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:06:20.109200 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:20.109174    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:06:20.392886 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:20.392795    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:06:20.408956 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:20.408928    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:06:23.293916 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:23.293860    2565 generic.go:356] "Generic (PLEG): container finished" podID="2d1929ff-8377-47e1-93cd-6d955cadc959" containerID="1786a3f4117071cdca2bd596dd6a23ce0db619a72b1108870aca3131adc42499" exitCode=0
Jul 08 17:06:23.293916 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:23.293899    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx" event={"ID":"2d1929ff-8377-47e1-93cd-6d955cadc959","Type":"ContainerDied","Data":"1786a3f4117071cdca2bd596dd6a23ce0db619a72b1108870aca3131adc42499"}
Jul 08 17:06:24.301691 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:24.301576    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx" event={"ID":"2d1929ff-8377-47e1-93cd-6d955cadc959","Type":"ContainerStarted","Data":"216963cfcde9b1f50c894154327d7c6645898fc60ebe1bde143159d7d892ca19"}
Jul 08 17:06:24.314559 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:24.314531    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:06:24.335282 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:24.335231    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx" podStartSLOduration=3.548071138 podStartE2EDuration="9.335203051s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.787131913s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:06:24.333344717 +0000 UTC m=+1921.046706147" watchObservedRunningTime="2026-07-08 17:06:24.335203051 +0000 UTC m=+1921.048564478"
Jul 08 17:06:25.369244 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:25.369200    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:06:26.315137 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:26.315103    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:06:26.316631 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:26.316602    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx" podUID="2d1929ff-8377-47e1-93cd-6d955cadc959" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 08 17:06:34.949688 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:34.949633    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:06:34.971373 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:34.971335    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"]
Jul 08 17:06:34.979286 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:34.979256    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"]
Jul 08 17:06:35.015801 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:35.015759    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:06:36.315273 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:36.315222    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx" podUID="2d1929ff-8377-47e1-93cd-6d955cadc959" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 08 17:06:46.314767 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:46.314714    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx" podUID="2d1929ff-8377-47e1-93cd-6d955cadc959" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 08 17:06:49.996016 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:49.995959    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv" podUID="e727c0f5-7c04-44bc-9204-a826f79d6587" containerName="main" containerID="cri-o://7085a4378028acf88c7f6dc1408b50b3149331ec7c73c49556e99ad6e62f5b93" gracePeriod=45
Jul 08 17:06:50.005581 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:50.005537    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k" podUID="b294d5b0-37b6-428a-b05c-dee4cf6d4e75" containerName="main" containerID="cri-o://1fd716e714e7d51155e5580241076f6e30785d6747dbba638ffe7302b629109c" gracePeriod=45
Jul 08 17:06:50.101651 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:50.101607    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv" podUID="e727c0f5-7c04-44bc-9204-a826f79d6587" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/health\": dial tcp 10.133.0.60:8000: connect: connection refused"
Jul 08 17:06:50.816031 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:50.815991    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k" podUID="b294d5b0-37b6-428a-b05c-dee4cf6d4e75" containerName="llm-d-routing-sidecar" containerID="cri-o://58d1c4cbf5d3e3b5b981fca145d486a9bb0bb9b3c53b28f4d36d52d018f25ef8" gracePeriod=45
Jul 08 17:06:50.951698 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:50.951654    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:06:51.071773 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.071745    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"e727c0f5-7c04-44bc-9204-a826f79d6587\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") "
Jul 08 17:06:51.072110 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.071836    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e727c0f5-7c04-44bc-9204-a826f79d6587-dshm\") pod \"e727c0f5-7c04-44bc-9204-a826f79d6587\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") "
Jul 08 17:06:51.072110 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.071879    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e727c0f5-7c04-44bc-9204-a826f79d6587-tmp-dir\") pod \"e727c0f5-7c04-44bc-9204-a826f79d6587\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") "
Jul 08 17:06:51.072110 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.071907    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dfjz\" (UniqueName: \"kubernetes.io/projected/e727c0f5-7c04-44bc-9204-a826f79d6587-kube-api-access-2dfjz\") pod \"e727c0f5-7c04-44bc-9204-a826f79d6587\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") "
Jul 08 17:06:51.072110 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.071955    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e727c0f5-7c04-44bc-9204-a826f79d6587-home\") pod \"e727c0f5-7c04-44bc-9204-a826f79d6587\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") "
Jul 08 17:06:51.072110 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.071986    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e727c0f5-7c04-44bc-9204-a826f79d6587-tls-certs\") pod \"e727c0f5-7c04-44bc-9204-a826f79d6587\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") "
Jul 08 17:06:51.072110 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.072019    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e727c0f5-7c04-44bc-9204-a826f79d6587-model-cache\") pod \"e727c0f5-7c04-44bc-9204-a826f79d6587\" (UID: \"e727c0f5-7c04-44bc-9204-a826f79d6587\") "
Jul 08 17:06:51.072598 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.072560    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e727c0f5-7c04-44bc-9204-a826f79d6587-model-cache" (OuterVolumeSpecName: "model-cache") pod "e727c0f5-7c04-44bc-9204-a826f79d6587" (UID: "e727c0f5-7c04-44bc-9204-a826f79d6587"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:06:51.073014 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.072845    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e727c0f5-7c04-44bc-9204-a826f79d6587-home" (OuterVolumeSpecName: "home") pod "e727c0f5-7c04-44bc-9204-a826f79d6587" (UID: "e727c0f5-7c04-44bc-9204-a826f79d6587"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:06:51.074446 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.074410    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e727c0f5-7c04-44bc-9204-a826f79d6587-dshm" (OuterVolumeSpecName: "dshm") pod "e727c0f5-7c04-44bc-9204-a826f79d6587" (UID: "e727c0f5-7c04-44bc-9204-a826f79d6587"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:06:51.074850 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.074810    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e727c0f5-7c04-44bc-9204-a826f79d6587-kube-api-access-2dfjz" (OuterVolumeSpecName: "kube-api-access-2dfjz") pod "e727c0f5-7c04-44bc-9204-a826f79d6587" (UID: "e727c0f5-7c04-44bc-9204-a826f79d6587"). InnerVolumeSpecName "kube-api-access-2dfjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:06:51.074951 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.074853    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e727c0f5-7c04-44bc-9204-a826f79d6587-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e727c0f5-7c04-44bc-9204-a826f79d6587" (UID: "e727c0f5-7c04-44bc-9204-a826f79d6587"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:06:51.075694 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.075653    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75" (OuterVolumeSpecName: "kserve-pvc-source") pod "e727c0f5-7c04-44bc-9204-a826f79d6587" (UID: "e727c0f5-7c04-44bc-9204-a826f79d6587"). InnerVolumeSpecName "pvc-6394c2ef-5d09-4573-b34c-dae5966e5434". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 08 17:06:51.076870 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.076853    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:06:51.083470 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.083430    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e727c0f5-7c04-44bc-9204-a826f79d6587-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e727c0f5-7c04-44bc-9204-a826f79d6587" (UID: "e727c0f5-7c04-44bc-9204-a826f79d6587"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:06:51.173526 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.173496    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") "
Jul 08 17:06:51.173763 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.173544    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-tls-certs\") pod \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") "
Jul 08 17:06:51.173763 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.173580    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mgjw\" (UniqueName: \"kubernetes.io/projected/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-kube-api-access-7mgjw\") pod \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") "
Jul 08 17:06:51.173763 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.173603    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-home\") pod \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") "
Jul 08 17:06:51.173763 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.173700    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-tmp-dir\") pod \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") "
Jul 08 17:06:51.173763 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.173730    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-model-cache\") pod \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") "
Jul 08 17:06:51.174039 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.173770    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-dshm\") pod \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\" (UID: \"b294d5b0-37b6-428a-b05c-dee4cf6d4e75\") "
Jul 08 17:06:51.174096 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.174046    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e727c0f5-7c04-44bc-9204-a826f79d6587-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:06:51.174096 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.174063    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e727c0f5-7c04-44bc-9204-a826f79d6587-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:06:51.174096 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.174079    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dfjz\" (UniqueName: \"kubernetes.io/projected/e727c0f5-7c04-44bc-9204-a826f79d6587-kube-api-access-2dfjz\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:06:51.174096 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.174092    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e727c0f5-7c04-44bc-9204-a826f79d6587-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:06:51.174304 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.174106    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e727c0f5-7c04-44bc-9204-a826f79d6587-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:06:51.174304 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.174120    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e727c0f5-7c04-44bc-9204-a826f79d6587-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:06:51.174304 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.174148    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-model-cache" (OuterVolumeSpecName: "model-cache") pod "b294d5b0-37b6-428a-b05c-dee4cf6d4e75" (UID: "b294d5b0-37b6-428a-b05c-dee4cf6d4e75"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:06:51.174488 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.174419    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-home" (OuterVolumeSpecName: "home") pod "b294d5b0-37b6-428a-b05c-dee4cf6d4e75" (UID: "b294d5b0-37b6-428a-b05c-dee4cf6d4e75"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:06:51.176163 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.176136    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b294d5b0-37b6-428a-b05c-dee4cf6d4e75" (UID: "b294d5b0-37b6-428a-b05c-dee4cf6d4e75"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:06:51.176163 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.176143    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-kube-api-access-7mgjw" (OuterVolumeSpecName: "kube-api-access-7mgjw") pod "b294d5b0-37b6-428a-b05c-dee4cf6d4e75" (UID: "b294d5b0-37b6-428a-b05c-dee4cf6d4e75"). InnerVolumeSpecName "kube-api-access-7mgjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:06:51.176335 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.176216    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-dshm" (OuterVolumeSpecName: "dshm") pod "b294d5b0-37b6-428a-b05c-dee4cf6d4e75" (UID: "b294d5b0-37b6-428a-b05c-dee4cf6d4e75"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:06:51.176970 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.176951    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75" (OuterVolumeSpecName: "kserve-pvc-source") pod "b294d5b0-37b6-428a-b05c-dee4cf6d4e75" (UID: "b294d5b0-37b6-428a-b05c-dee4cf6d4e75"). InnerVolumeSpecName "pvc-6394c2ef-5d09-4573-b34c-dae5966e5434". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 08 17:06:51.194326 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.194305    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b294d5b0-37b6-428a-b05c-dee4cf6d4e75" (UID: "b294d5b0-37b6-428a-b05c-dee4cf6d4e75"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:06:51.275458 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.275355    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mgjw\" (UniqueName: \"kubernetes.io/projected/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-kube-api-access-7mgjw\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:06:51.275458 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.275388    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:06:51.275458 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.275403    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:06:51.275458 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.275415    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:06:51.275458 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.275427    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:06:51.275858 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.275462    2565 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") on node \"ip-10-0-128-142.ec2.internal\" "
Jul 08 17:06:51.275858 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.275477    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b294d5b0-37b6-428a-b05c-dee4cf6d4e75-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:06:51.304131 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.304105    2565 operation_generator.go:895] UnmountDevice succeeded for volume "pvc-6394c2ef-5d09-4573-b34c-dae5966e5434" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75") on node "ip-10-0-128-142.ec2.internal"
Jul 08 17:06:51.376265 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.376222    2565 reconciler_common.go:299] "Volume detached for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"csi-2dacefae46c7a149c270f1437d3b1a0444209eeb1f65f5a21ee1f03bbc64e3c5\""
Jul 08 17:06:51.464610 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.464573    2565 generic.go:356] "Generic (PLEG): container finished" podID="b294d5b0-37b6-428a-b05c-dee4cf6d4e75" containerID="1fd716e714e7d51155e5580241076f6e30785d6747dbba638ffe7302b629109c" exitCode=0
Jul 08 17:06:51.464610 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.464601    2565 generic.go:356] "Generic (PLEG): container finished" podID="b294d5b0-37b6-428a-b05c-dee4cf6d4e75" containerID="58d1c4cbf5d3e3b5b981fca145d486a9bb0bb9b3c53b28f4d36d52d018f25ef8" exitCode=0
Jul 08 17:06:51.464863 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.464627    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"
Jul 08 17:06:51.464863 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.464665    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k" event={"ID":"b294d5b0-37b6-428a-b05c-dee4cf6d4e75","Type":"ContainerDied","Data":"1fd716e714e7d51155e5580241076f6e30785d6747dbba638ffe7302b629109c"}
Jul 08 17:06:51.464863 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.464783    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k" event={"ID":"b294d5b0-37b6-428a-b05c-dee4cf6d4e75","Type":"ContainerDied","Data":"58d1c4cbf5d3e3b5b981fca145d486a9bb0bb9b3c53b28f4d36d52d018f25ef8"}
Jul 08 17:06:51.464863 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.464813    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k" event={"ID":"b294d5b0-37b6-428a-b05c-dee4cf6d4e75","Type":"ContainerDied","Data":"022894c8d472aeb975c573bfc0a217aff825dc4b9bcc5849edc9e5379f1aedd0"}
Jul 08 17:06:51.464863 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.464847    2565 scope.go:117] "RemoveContainer" containerID="1fd716e714e7d51155e5580241076f6e30785d6747dbba638ffe7302b629109c"
Jul 08 17:06:51.466216 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.466195    2565 generic.go:356] "Generic (PLEG): container finished" podID="e727c0f5-7c04-44bc-9204-a826f79d6587" containerID="7085a4378028acf88c7f6dc1408b50b3149331ec7c73c49556e99ad6e62f5b93" exitCode=0
Jul 08 17:06:51.466324 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.466242    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"
Jul 08 17:06:51.466324 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.466269    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv" event={"ID":"e727c0f5-7c04-44bc-9204-a826f79d6587","Type":"ContainerDied","Data":"7085a4378028acf88c7f6dc1408b50b3149331ec7c73c49556e99ad6e62f5b93"}
Jul 08 17:06:51.466324 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.466307    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv" event={"ID":"e727c0f5-7c04-44bc-9204-a826f79d6587","Type":"ContainerDied","Data":"e7a25967c25c081b051905be4e64bed76317a600d63efa19d1341951781ed92e"}
Jul 08 17:06:51.475762 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.475738    2565 scope.go:117] "RemoveContainer" containerID="58d1c4cbf5d3e3b5b981fca145d486a9bb0bb9b3c53b28f4d36d52d018f25ef8"
Jul 08 17:06:51.484825 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.484800    2565 scope.go:117] "RemoveContainer" containerID="1fd716e714e7d51155e5580241076f6e30785d6747dbba638ffe7302b629109c"
Jul 08 17:06:51.485123 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:06:51.485102    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fd716e714e7d51155e5580241076f6e30785d6747dbba638ffe7302b629109c\": container with ID starting with 1fd716e714e7d51155e5580241076f6e30785d6747dbba638ffe7302b629109c not found: ID does not exist" containerID="1fd716e714e7d51155e5580241076f6e30785d6747dbba638ffe7302b629109c"
Jul 08 17:06:51.485206 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.485134    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fd716e714e7d51155e5580241076f6e30785d6747dbba638ffe7302b629109c"} err="failed to get container status \"1fd716e714e7d51155e5580241076f6e30785d6747dbba638ffe7302b629109c\": rpc error: code = NotFound desc = could not find container \"1fd716e714e7d51155e5580241076f6e30785d6747dbba638ffe7302b629109c\": container with ID starting with 1fd716e714e7d51155e5580241076f6e30785d6747dbba638ffe7302b629109c not found: ID does not exist"
Jul 08 17:06:51.485206 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.485157    2565 scope.go:117] "RemoveContainer" containerID="58d1c4cbf5d3e3b5b981fca145d486a9bb0bb9b3c53b28f4d36d52d018f25ef8"
Jul 08 17:06:51.485450 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:06:51.485426    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58d1c4cbf5d3e3b5b981fca145d486a9bb0bb9b3c53b28f4d36d52d018f25ef8\": container with ID starting with 58d1c4cbf5d3e3b5b981fca145d486a9bb0bb9b3c53b28f4d36d52d018f25ef8 not found: ID does not exist" containerID="58d1c4cbf5d3e3b5b981fca145d486a9bb0bb9b3c53b28f4d36d52d018f25ef8"
Jul 08 17:06:51.485491 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.485463    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58d1c4cbf5d3e3b5b981fca145d486a9bb0bb9b3c53b28f4d36d52d018f25ef8"} err="failed to get container status \"58d1c4cbf5d3e3b5b981fca145d486a9bb0bb9b3c53b28f4d36d52d018f25ef8\": rpc error: code = NotFound desc = could not find container \"58d1c4cbf5d3e3b5b981fca145d486a9bb0bb9b3c53b28f4d36d52d018f25ef8\": container with ID starting with 58d1c4cbf5d3e3b5b981fca145d486a9bb0bb9b3c53b28f4d36d52d018f25ef8 not found: ID does not exist"
Jul 08 17:06:51.485491 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.485485    2565 scope.go:117] "RemoveContainer" containerID="1fd716e714e7d51155e5580241076f6e30785d6747dbba638ffe7302b629109c"
Jul 08 17:06:51.485741 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.485722    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fd716e714e7d51155e5580241076f6e30785d6747dbba638ffe7302b629109c"} err="failed to get container status \"1fd716e714e7d51155e5580241076f6e30785d6747dbba638ffe7302b629109c\": rpc error: code = NotFound desc = could not find container \"1fd716e714e7d51155e5580241076f6e30785d6747dbba638ffe7302b629109c\": container with ID starting with 1fd716e714e7d51155e5580241076f6e30785d6747dbba638ffe7302b629109c not found: ID does not exist"
Jul 08 17:06:51.485828 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.485743    2565 scope.go:117] "RemoveContainer" containerID="58d1c4cbf5d3e3b5b981fca145d486a9bb0bb9b3c53b28f4d36d52d018f25ef8"
Jul 08 17:06:51.485967 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.485947    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58d1c4cbf5d3e3b5b981fca145d486a9bb0bb9b3c53b28f4d36d52d018f25ef8"} err="failed to get container status \"58d1c4cbf5d3e3b5b981fca145d486a9bb0bb9b3c53b28f4d36d52d018f25ef8\": rpc error: code = NotFound desc = could not find container \"58d1c4cbf5d3e3b5b981fca145d486a9bb0bb9b3c53b28f4d36d52d018f25ef8\": container with ID starting with 58d1c4cbf5d3e3b5b981fca145d486a9bb0bb9b3c53b28f4d36d52d018f25ef8 not found: ID does not exist"
Jul 08 17:06:51.486035 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.485968    2565 scope.go:117] "RemoveContainer" containerID="7085a4378028acf88c7f6dc1408b50b3149331ec7c73c49556e99ad6e62f5b93"
Jul 08 17:06:51.493981 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.493959    2565 scope.go:117] "RemoveContainer" containerID="7085a4378028acf88c7f6dc1408b50b3149331ec7c73c49556e99ad6e62f5b93"
Jul 08 17:06:51.494258 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:06:51.494240    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7085a4378028acf88c7f6dc1408b50b3149331ec7c73c49556e99ad6e62f5b93\": container with ID starting with 7085a4378028acf88c7f6dc1408b50b3149331ec7c73c49556e99ad6e62f5b93 not found: ID does not exist" containerID="7085a4378028acf88c7f6dc1408b50b3149331ec7c73c49556e99ad6e62f5b93"
Jul 08 17:06:51.494305 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.494264    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7085a4378028acf88c7f6dc1408b50b3149331ec7c73c49556e99ad6e62f5b93"} err="failed to get container status \"7085a4378028acf88c7f6dc1408b50b3149331ec7c73c49556e99ad6e62f5b93\": rpc error: code = NotFound desc = could not find container \"7085a4378028acf88c7f6dc1408b50b3149331ec7c73c49556e99ad6e62f5b93\": container with ID starting with 7085a4378028acf88c7f6dc1408b50b3149331ec7c73c49556e99ad6e62f5b93 not found: ID does not exist"
Jul 08 17:06:51.504124 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.504099    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"]
Jul 08 17:06:51.511186 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.511163    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-7db79954c-2hv7k"]
Jul 08 17:06:51.530829 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.530749    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"]
Jul 08 17:06:51.536407 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.536379    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-e8706282-kserve-prefill-795tshv"]
Jul 08 17:06:51.822694 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.822594    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b294d5b0-37b6-428a-b05c-dee4cf6d4e75" path="/var/lib/kubelet/pods/b294d5b0-37b6-428a-b05c-dee4cf6d4e75/volumes"
Jul 08 17:06:51.823359 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:51.823335    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e727c0f5-7c04-44bc-9204-a826f79d6587" path="/var/lib/kubelet/pods/e727c0f5-7c04-44bc-9204-a826f79d6587/volumes"
Jul 08 17:06:56.314897 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:56.314855    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx" podUID="2d1929ff-8377-47e1-93cd-6d955cadc959" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 08 17:06:59.889595 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:06:59.889557    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:07:00.015506 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.015473    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:07:00.150006 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.149902    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:07:00.201843 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.201813    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 08 17:07:00.202230 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.202216    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b294d5b0-37b6-428a-b05c-dee4cf6d4e75" containerName="main"
Jul 08 17:07:00.202276 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.202232    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b294d5b0-37b6-428a-b05c-dee4cf6d4e75" containerName="main"
Jul 08 17:07:00.202276 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.202243    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b294d5b0-37b6-428a-b05c-dee4cf6d4e75" containerName="llm-d-routing-sidecar"
Jul 08 17:07:00.202276 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.202250    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="b294d5b0-37b6-428a-b05c-dee4cf6d4e75" containerName="llm-d-routing-sidecar"
Jul 08 17:07:00.202276 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.202260    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e727c0f5-7c04-44bc-9204-a826f79d6587" containerName="main"
Jul 08 17:07:00.202276 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.202265    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e727c0f5-7c04-44bc-9204-a826f79d6587" containerName="main"
Jul 08 17:07:00.202440 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.202345    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e727c0f5-7c04-44bc-9204-a826f79d6587" containerName="main"
Jul 08 17:07:00.202440 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.202365    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b294d5b0-37b6-428a-b05c-dee4cf6d4e75" containerName="main"
Jul 08 17:07:00.202440 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.202375    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="b294d5b0-37b6-428a-b05c-dee4cf6d4e75" containerName="llm-d-routing-sidecar"
Jul 08 17:07:00.204369 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.204352    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:07:00.206991 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.206969    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv1dc4269d1ada5f2d28562215d180c57f-kserve-self-signed-certs\""
Jul 08 17:07:00.208090 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.208064    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-pvc-router-manage-2577e7-b09f84b1-dockercfg-97lq6\""
Jul 08 17:07:00.219956 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.219925    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 08 17:07:00.259997 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.259965    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 08 17:07:00.261145 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.261127    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:07:00.261209 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.261174    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:07:00.261209 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.261199    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:07:00.261299 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.261217    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k9vj\" (UniqueName: \"kubernetes.io/projected/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-kube-api-access-8k9vj\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:07:00.261299 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.261239    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:07:00.261299 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.261268    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:07:00.263079 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.263060    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 17:07:00.275277 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.275254    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 08 17:07:00.362343 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.362313    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:07:00.362562 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.362382    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:07:00.362562 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.362431    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e7a242b-c2d7-49ae-8c07-eb8743793960-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0e7a242b-c2d7-49ae-8c07-eb8743793960\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 17:07:00.362562 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.362480    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:07:00.362786 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.362565    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e7a242b-c2d7-49ae-8c07-eb8743793960-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0e7a242b-c2d7-49ae-8c07-eb8743793960\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 17:07:00.362786 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.362608    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e7a242b-c2d7-49ae-8c07-eb8743793960-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0e7a242b-c2d7-49ae-8c07-eb8743793960\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 17:07:00.362786 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.362654    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:07:00.362786 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.362699    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k9vj\" (UniqueName: \"kubernetes.io/projected/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-kube-api-access-8k9vj\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:07:00.362786 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.362716    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:07:00.362786 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.362724    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r99gf\" (UniqueName: \"kubernetes.io/projected/0e7a242b-c2d7-49ae-8c07-eb8743793960-kube-api-access-r99gf\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0e7a242b-c2d7-49ae-8c07-eb8743793960\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 17:07:00.363120 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.362804    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e7a242b-c2d7-49ae-8c07-eb8743793960-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0e7a242b-c2d7-49ae-8c07-eb8743793960\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 17:07:00.363120 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.362835    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:07:00.363120 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.362885    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e7a242b-c2d7-49ae-8c07-eb8743793960-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0e7a242b-c2d7-49ae-8c07-eb8743793960\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 17:07:00.363120 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.363084    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:07:00.363339 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.363173    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:07:00.364924 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.364900    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:07:00.365540 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.365493    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:07:00.374530 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.374506    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k9vj\" (UniqueName: \"kubernetes.io/projected/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-kube-api-access-8k9vj\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:07:00.414052 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.413976    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:07:00.463861 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.463828    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e7a242b-c2d7-49ae-8c07-eb8743793960-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0e7a242b-c2d7-49ae-8c07-eb8743793960\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 17:07:00.464027 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.463884    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e7a242b-c2d7-49ae-8c07-eb8743793960-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0e7a242b-c2d7-49ae-8c07-eb8743793960\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 17:07:00.464027 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.463903    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e7a242b-c2d7-49ae-8c07-eb8743793960-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0e7a242b-c2d7-49ae-8c07-eb8743793960\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 17:07:00.464027 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.463928    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r99gf\" (UniqueName: \"kubernetes.io/projected/0e7a242b-c2d7-49ae-8c07-eb8743793960-kube-api-access-r99gf\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0e7a242b-c2d7-49ae-8c07-eb8743793960\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 17:07:00.464027 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.463955    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e7a242b-c2d7-49ae-8c07-eb8743793960-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0e7a242b-c2d7-49ae-8c07-eb8743793960\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 17:07:00.464027 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.463988    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e7a242b-c2d7-49ae-8c07-eb8743793960-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0e7a242b-c2d7-49ae-8c07-eb8743793960\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 17:07:00.464331 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.464303    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e7a242b-c2d7-49ae-8c07-eb8743793960-model-cache\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0e7a242b-c2d7-49ae-8c07-eb8743793960\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 17:07:00.464396 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.464345    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e7a242b-c2d7-49ae-8c07-eb8743793960-home\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0e7a242b-c2d7-49ae-8c07-eb8743793960\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 17:07:00.464396 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.464378    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e7a242b-c2d7-49ae-8c07-eb8743793960-tmp-dir\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0e7a242b-c2d7-49ae-8c07-eb8743793960\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 17:07:00.466021 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.465998    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e7a242b-c2d7-49ae-8c07-eb8743793960-dshm\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0e7a242b-c2d7-49ae-8c07-eb8743793960\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 17:07:00.466341 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.466325    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e7a242b-c2d7-49ae-8c07-eb8743793960-tls-certs\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0e7a242b-c2d7-49ae-8c07-eb8743793960\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 17:07:00.474520 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:00.474491    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r99gf\" (UniqueName: \"kubernetes.io/projected/0e7a242b-c2d7-49ae-8c07-eb8743793960-kube-api-access-r99gf\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0e7a242b-c2d7-49ae-8c07-eb8743793960\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 17:07:04.201761 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:04.201729    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:07:04.210732 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:04.210699    2565 operation_generator.go:1469] "Controller attach succeeded for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") device path: \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:07:04.302648 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:04.302612    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:07:04.302805 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:04.302759    2565 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") DevicePath \"\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:07:04.306832 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:04.306811    2565 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") DevicePath \"csi-2dacefae46c7a149c270f1437d3b1a0444209eeb1f65f5a21ee1f03bbc64e3c5\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:07:04.393324 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:04.393289    2565 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/d4c2afd44bc1329e2e7018951f0ca78b5032d86e3fdf6923f6d1397e5ac2aa83/globalmount\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:07:04.407851 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:04.407823    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:07:04.416340 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:04.416319    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:07:04.505360 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:04.505325    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0e7a242b-c2d7-49ae-8c07-eb8743793960\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 17:07:04.505505 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:04.505471    2565 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0e7a242b-c2d7-49ae-8c07-eb8743793960\") DevicePath \"csi-2dacefae46c7a149c270f1437d3b1a0444209eeb1f65f5a21ee1f03bbc64e3c5\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 17:07:04.509713 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:04.509651    2565 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0e7a242b-c2d7-49ae-8c07-eb8743793960\") DevicePath \"csi-2dacefae46c7a149c270f1437d3b1a0444209eeb1f65f5a21ee1f03bbc64e3c5\"" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 17:07:04.533981 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:04.533946    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1\" (UID: \"0e7a242b-c2d7-49ae-8c07-eb8743793960\") " pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 17:07:04.559180 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:04.559156    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 08 17:07:04.561007 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:07:04.560979    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d4f4f3_2d00_4f5a_ba45_d4e44f350af1.slice/crio-049841a98a3b829ba7f62fe51dd5ed5258cf90c718dd51c99172c9a04b7a51b2 WatchSource:0}: Error finding container 049841a98a3b829ba7f62fe51dd5ed5258cf90c718dd51c99172c9a04b7a51b2: Status 404 returned error can't find the container with id 049841a98a3b829ba7f62fe51dd5ed5258cf90c718dd51c99172c9a04b7a51b2
Jul 08 17:07:04.774567 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:04.774485    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 17:07:04.946957 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:04.946928    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 08 17:07:04.948376 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:07:04.948352    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e7a242b_c2d7_49ae_8c07_eb8743793960.slice/crio-545d2e0871d179d8a0d8531d3795b648d626181ec0bb336a03770f9cf3609974 WatchSource:0}: Error finding container 545d2e0871d179d8a0d8531d3795b648d626181ec0bb336a03770f9cf3609974: Status 404 returned error can't find the container with id 545d2e0871d179d8a0d8531d3795b648d626181ec0bb336a03770f9cf3609974
Jul 08 17:07:05.549731 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:05.549686    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1","Type":"ContainerStarted","Data":"07ff61abcb486c810e8a209de4fe8a61c14bd022d34846bafb2e75eacedd5840"}
Jul 08 17:07:05.550202 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:05.549761    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1","Type":"ContainerStarted","Data":"049841a98a3b829ba7f62fe51dd5ed5258cf90c718dd51c99172c9a04b7a51b2"}
Jul 08 17:07:05.551414 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:05.551389    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"0e7a242b-c2d7-49ae-8c07-eb8743793960","Type":"ContainerStarted","Data":"2bcc12d7127e510c6c1d51753d44897665410472499901d94cb074d075d116e9"}
Jul 08 17:07:05.551561 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:05.551436    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"0e7a242b-c2d7-49ae-8c07-eb8743793960","Type":"ContainerStarted","Data":"545d2e0871d179d8a0d8531d3795b648d626181ec0bb336a03770f9cf3609974"}
Jul 08 17:07:05.582787 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:05.582729    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podStartSLOduration=5.582707347 podStartE2EDuration="5.582707347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:07:05.578546586 +0000 UTC m=+1962.291908049" watchObservedRunningTime="2026-07-08 17:07:05.582707347 +0000 UTC m=+1962.296068774"
Jul 08 17:07:05.612062 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:05.612010    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podStartSLOduration=5.611993712 podStartE2EDuration="5.611993712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:07:05.608254196 +0000 UTC m=+1962.321615641" watchObservedRunningTime="2026-07-08 17:07:05.611993712 +0000 UTC m=+1962.325355144"
Jul 08 17:07:06.315014 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:06.314960    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx" podUID="2d1929ff-8377-47e1-93cd-6d955cadc959" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 08 17:07:06.416453 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:06.416420    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:07:11.070236 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:11.070199    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:07:14.416512 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:14.416464    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:07:14.418587 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:14.418537    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 08 17:07:16.315171 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:16.315053    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx" podUID="2d1929ff-8377-47e1-93cd-6d955cadc959" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 08 17:07:20.023098 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:20.023055    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:07:24.417565 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:24.417455    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 08 17:07:26.315451 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:26.315399    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx" podUID="2d1929ff-8377-47e1-93cd-6d955cadc959" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 08 17:07:34.417074 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:34.417020    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 08 17:07:36.315375 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:36.315326    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx" podUID="2d1929ff-8377-47e1-93cd-6d955cadc959" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 08 17:07:44.417100 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:44.417050    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 08 17:07:46.314947 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:46.314898    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx" podUID="2d1929ff-8377-47e1-93cd-6d955cadc959" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 08 17:07:54.417147 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:54.417101    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 08 17:07:56.315007 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:07:56.314962    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx" podUID="2d1929ff-8377-47e1-93cd-6d955cadc959" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 08 17:08:04.416873 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:08:04.416819    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 08 17:08:06.315647 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:08:06.315604    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx" podUID="2d1929ff-8377-47e1-93cd-6d955cadc959" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 08 17:08:14.416818 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:08:14.416775    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 08 17:08:16.315562 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:08:16.315514    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx" podUID="2d1929ff-8377-47e1-93cd-6d955cadc959" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 08 17:08:24.417325 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:08:24.417273    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 08 17:08:26.315289 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:08:26.315238    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx" podUID="2d1929ff-8377-47e1-93cd-6d955cadc959" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 08 17:08:34.417646 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:08:34.417597    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 08 17:08:36.315334 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:08:36.315290    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx" podUID="2d1929ff-8377-47e1-93cd-6d955cadc959" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 08 17:08:44.416865 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:08:44.416824    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 08 17:08:46.315206 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:08:46.315106    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx" podUID="2d1929ff-8377-47e1-93cd-6d955cadc959" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 08 17:08:54.417321 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:08:54.417262    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 08 17:08:56.325735 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:08:56.325700    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:08:56.333974 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:08:56.333935    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:09:02.151642 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:02.151592    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:09:02.219298 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:02.219264    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:09:04.417497 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:04.417450    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 08 17:09:09.078890 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:09.078849    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:09:09.085452 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:09.085409    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"]
Jul 08 17:09:09.136560 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:09.136505    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:09:14.417044 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:14.416999    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 08 17:09:24.111951 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:24.111891    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx" podUID="2d1929ff-8377-47e1-93cd-6d955cadc959" containerName="main" containerID="cri-o://216963cfcde9b1f50c894154327d7c6645898fc60ebe1bde143159d7d892ca19" gracePeriod=45
Jul 08 17:09:24.327171 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:24.327120    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx" podUID="2d1929ff-8377-47e1-93cd-6d955cadc959" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 08 17:09:24.416945 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:24.416847    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 08 17:09:25.326396 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:25.326361    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx" podUID="2d1929ff-8377-47e1-93cd-6d955cadc959" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": dial tcp 10.133.0.61:8000: connect: connection refused"
Jul 08 17:09:25.326647 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:25.326491    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:09:25.429066 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:25.429030    2565 generic.go:356] "Generic (PLEG): container finished" podID="2d1929ff-8377-47e1-93cd-6d955cadc959" containerID="216963cfcde9b1f50c894154327d7c6645898fc60ebe1bde143159d7d892ca19" exitCode=0
Jul 08 17:09:25.429230 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:25.429062    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx" event={"ID":"2d1929ff-8377-47e1-93cd-6d955cadc959","Type":"ContainerDied","Data":"216963cfcde9b1f50c894154327d7c6645898fc60ebe1bde143159d7d892ca19"}
Jul 08 17:09:25.484572 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:25.484545    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:09:25.608403 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:25.608307    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-tmp-dir\") pod \"2d1929ff-8377-47e1-93cd-6d955cadc959\" (UID: \"2d1929ff-8377-47e1-93cd-6d955cadc959\") "
Jul 08 17:09:25.608403 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:25.608367    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xdwm7\" (UniqueName: \"kubernetes.io/projected/2d1929ff-8377-47e1-93cd-6d955cadc959-kube-api-access-xdwm7\") pod \"2d1929ff-8377-47e1-93cd-6d955cadc959\" (UID: \"2d1929ff-8377-47e1-93cd-6d955cadc959\") "
Jul 08 17:09:25.608403 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:25.608391    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-dshm\") pod \"2d1929ff-8377-47e1-93cd-6d955cadc959\" (UID: \"2d1929ff-8377-47e1-93cd-6d955cadc959\") "
Jul 08 17:09:25.608723 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:25.608413    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-kserve-provision-location\") pod \"2d1929ff-8377-47e1-93cd-6d955cadc959\" (UID: \"2d1929ff-8377-47e1-93cd-6d955cadc959\") "
Jul 08 17:09:25.608723 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:25.608453    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1929ff-8377-47e1-93cd-6d955cadc959-tls-certs\") pod \"2d1929ff-8377-47e1-93cd-6d955cadc959\" (UID: \"2d1929ff-8377-47e1-93cd-6d955cadc959\") "
Jul 08 17:09:25.608723 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:25.608486    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-home\") pod \"2d1929ff-8377-47e1-93cd-6d955cadc959\" (UID: \"2d1929ff-8377-47e1-93cd-6d955cadc959\") "
Jul 08 17:09:25.608723 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:25.608523    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-model-cache\") pod \"2d1929ff-8377-47e1-93cd-6d955cadc959\" (UID: \"2d1929ff-8377-47e1-93cd-6d955cadc959\") "
Jul 08 17:09:25.608965 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:25.608931    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-model-cache" (OuterVolumeSpecName: "model-cache") pod "2d1929ff-8377-47e1-93cd-6d955cadc959" (UID: "2d1929ff-8377-47e1-93cd-6d955cadc959"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:09:25.609427 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:25.609401    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-home" (OuterVolumeSpecName: "home") pod "2d1929ff-8377-47e1-93cd-6d955cadc959" (UID: "2d1929ff-8377-47e1-93cd-6d955cadc959"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:09:25.610801 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:25.610776    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-dshm" (OuterVolumeSpecName: "dshm") pod "2d1929ff-8377-47e1-93cd-6d955cadc959" (UID: "2d1929ff-8377-47e1-93cd-6d955cadc959"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:09:25.611103 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:25.611058    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1929ff-8377-47e1-93cd-6d955cadc959-kube-api-access-xdwm7" (OuterVolumeSpecName: "kube-api-access-xdwm7") pod "2d1929ff-8377-47e1-93cd-6d955cadc959" (UID: "2d1929ff-8377-47e1-93cd-6d955cadc959"). InnerVolumeSpecName "kube-api-access-xdwm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:09:25.611360 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:25.611336    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d1929ff-8377-47e1-93cd-6d955cadc959-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d1929ff-8377-47e1-93cd-6d955cadc959" (UID: "2d1929ff-8377-47e1-93cd-6d955cadc959"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:09:25.622510 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:25.622486    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2d1929ff-8377-47e1-93cd-6d955cadc959" (UID: "2d1929ff-8377-47e1-93cd-6d955cadc959"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:09:25.707296 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:25.707248    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2d1929ff-8377-47e1-93cd-6d955cadc959" (UID: "2d1929ff-8377-47e1-93cd-6d955cadc959"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:09:25.709309 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:25.709288    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1929ff-8377-47e1-93cd-6d955cadc959-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:09:25.709383 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:25.709311    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:09:25.709383 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:25.709320    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:09:25.709383 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:25.709329    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:09:25.709383 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:25.709337    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdwm7\" (UniqueName: \"kubernetes.io/projected/2d1929ff-8377-47e1-93cd-6d955cadc959-kube-api-access-xdwm7\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:09:25.709383 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:25.709345    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:09:25.709383 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:25.709355    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2d1929ff-8377-47e1-93cd-6d955cadc959-kserve-provision-location\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:09:26.436881 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:26.436840    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx" event={"ID":"2d1929ff-8377-47e1-93cd-6d955cadc959","Type":"ContainerDied","Data":"33d206715d2eb3e908a99492ab12fdf77773490a6260ec53066dea3c627c0dad"}
Jul 08 17:09:26.436881 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:26.436860    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"
Jul 08 17:09:26.437399 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:26.436933    2565 scope.go:117] "RemoveContainer" containerID="216963cfcde9b1f50c894154327d7c6645898fc60ebe1bde143159d7d892ca19"
Jul 08 17:09:26.447793 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:26.447774    2565 scope.go:117] "RemoveContainer" containerID="1786a3f4117071cdca2bd596dd6a23ce0db619a72b1108870aca3131adc42499"
Jul 08 17:09:26.460294 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:26.460267    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"]
Jul 08 17:09:26.464736 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:26.464705    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-test-kserve-86c9c75755-7p6kx"]
Jul 08 17:09:27.823266 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:27.823220    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d1929ff-8377-47e1-93cd-6d955cadc959" path="/var/lib/kubelet/pods/2d1929ff-8377-47e1-93cd-6d955cadc959/volumes"
Jul 08 17:09:34.417051 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:34.417004    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 08 17:09:43.875934 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:43.875897    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"]
Jul 08 17:09:43.876558 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:43.876511    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d1929ff-8377-47e1-93cd-6d955cadc959" containerName="storage-initializer"
Jul 08 17:09:43.876558 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:43.876549    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d1929ff-8377-47e1-93cd-6d955cadc959" containerName="storage-initializer"
Jul 08 17:09:43.876767 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:43.876599    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d1929ff-8377-47e1-93cd-6d955cadc959" containerName="main"
Jul 08 17:09:43.876767 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:43.876608    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d1929ff-8377-47e1-93cd-6d955cadc959" containerName="main"
Jul 08 17:09:43.876767 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:43.876727    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d1929ff-8377-47e1-93cd-6d955cadc959" containerName="main"
Jul 08 17:09:43.881896 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:43.881861    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"
Jul 08 17:09:43.884768 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:43.884438    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-test-kserve-self-signed-certs\""
Jul 08 17:09:43.884768 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:43.884529    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 17:09:43.891751 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:43.891728    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"]
Jul 08 17:09:43.914136 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:43.914053    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:09:43.995647 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:43.995599    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-dshm\") pod \"router-with-refs-test-kserve-697958d7-hhdqd\" (UID: \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"
Jul 08 17:09:43.995647 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:43.995647    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-tmp-dir\") pod \"router-with-refs-test-kserve-697958d7-hhdqd\" (UID: \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"
Jul 08 17:09:43.995928 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:43.995666    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-tls-certs\") pod \"router-with-refs-test-kserve-697958d7-hhdqd\" (UID: \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"
Jul 08 17:09:43.995928 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:43.995752    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-kserve-provision-location\") pod \"router-with-refs-test-kserve-697958d7-hhdqd\" (UID: \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"
Jul 08 17:09:43.995928 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:43.995796    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-model-cache\") pod \"router-with-refs-test-kserve-697958d7-hhdqd\" (UID: \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"
Jul 08 17:09:43.995928 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:43.995829    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctwgj\" (UniqueName: \"kubernetes.io/projected/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-kube-api-access-ctwgj\") pod \"router-with-refs-test-kserve-697958d7-hhdqd\" (UID: \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"
Jul 08 17:09:43.996116 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:43.995978    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-home\") pod \"router-with-refs-test-kserve-697958d7-hhdqd\" (UID: \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"
Jul 08 17:09:44.097121 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:44.097073    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctwgj\" (UniqueName: \"kubernetes.io/projected/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-kube-api-access-ctwgj\") pod \"router-with-refs-test-kserve-697958d7-hhdqd\" (UID: \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"
Jul 08 17:09:44.097326 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:44.097189    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-home\") pod \"router-with-refs-test-kserve-697958d7-hhdqd\" (UID: \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"
Jul 08 17:09:44.097326 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:44.097214    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-dshm\") pod \"router-with-refs-test-kserve-697958d7-hhdqd\" (UID: \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"
Jul 08 17:09:44.097326 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:44.097240    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-tmp-dir\") pod \"router-with-refs-test-kserve-697958d7-hhdqd\" (UID: \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"
Jul 08 17:09:44.097326 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:44.097257    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-tls-certs\") pod \"router-with-refs-test-kserve-697958d7-hhdqd\" (UID: \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"
Jul 08 17:09:44.097326 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:44.097281    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-kserve-provision-location\") pod \"router-with-refs-test-kserve-697958d7-hhdqd\" (UID: \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"
Jul 08 17:09:44.097326 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:44.097310    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-model-cache\") pod \"router-with-refs-test-kserve-697958d7-hhdqd\" (UID: \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"
Jul 08 17:09:44.097789 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:44.097762    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-home\") pod \"router-with-refs-test-kserve-697958d7-hhdqd\" (UID: \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"
Jul 08 17:09:44.097896 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:44.097795    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-model-cache\") pod \"router-with-refs-test-kserve-697958d7-hhdqd\" (UID: \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"
Jul 08 17:09:44.097896 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:44.097833    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-tmp-dir\") pod \"router-with-refs-test-kserve-697958d7-hhdqd\" (UID: \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"
Jul 08 17:09:44.098235 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:44.098201    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-kserve-provision-location\") pod \"router-with-refs-test-kserve-697958d7-hhdqd\" (UID: \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"
Jul 08 17:09:44.100029 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:44.100010    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-dshm\") pod \"router-with-refs-test-kserve-697958d7-hhdqd\" (UID: \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"
Jul 08 17:09:44.100399 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:44.100378    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-tls-certs\") pod \"router-with-refs-test-kserve-697958d7-hhdqd\" (UID: \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"
Jul 08 17:09:44.107047 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:44.107023    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctwgj\" (UniqueName: \"kubernetes.io/projected/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-kube-api-access-ctwgj\") pod \"router-with-refs-test-kserve-697958d7-hhdqd\" (UID: \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\") " pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"
Jul 08 17:09:44.197775 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:44.197737    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"
Jul 08 17:09:44.352347 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:44.352321    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"]
Jul 08 17:09:44.355248 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:09:44.355205    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc3ac9f3_d750_4cd9_a44e_32df8afd5e9a.slice/crio-49f64f18f8cf983de2219a5016358719855dc96d3ae5d226340c79c162ee9671 WatchSource:0}: Error finding container 49f64f18f8cf983de2219a5016358719855dc96d3ae5d226340c79c162ee9671: Status 404 returned error can't find the container with id 49f64f18f8cf983de2219a5016358719855dc96d3ae5d226340c79c162ee9671
Jul 08 17:09:44.417810 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:44.417773    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 08 17:09:44.554460 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:44.554311    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd" event={"ID":"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a","Type":"ContainerStarted","Data":"cf3d505357d8d59ead8438c1ce28cf8f2c5001f3c8524af10dd0d9d3031ee6ab"}
Jul 08 17:09:44.554460 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:44.554386    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd" event={"ID":"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a","Type":"ContainerStarted","Data":"49f64f18f8cf983de2219a5016358719855dc96d3ae5d226340c79c162ee9671"}
Jul 08 17:09:44.581869 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:44.581834    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:09:44.680807 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:44.680777    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:09:53.605397 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:53.605366    2565 generic.go:356] "Generic (PLEG): container finished" podID="bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a" containerID="cf3d505357d8d59ead8438c1ce28cf8f2c5001f3c8524af10dd0d9d3031ee6ab" exitCode=0
Jul 08 17:09:53.605745 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:53.605437    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd" event={"ID":"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a","Type":"ContainerDied","Data":"cf3d505357d8d59ead8438c1ce28cf8f2c5001f3c8524af10dd0d9d3031ee6ab"}
Jul 08 17:09:54.427349 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:54.427320    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:09:54.436897 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:54.436868    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:09:54.613219 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:54.613185    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd" event={"ID":"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a","Type":"ContainerStarted","Data":"a8c23f447813033467af1361dcf397ae55a18dea788e0f84892dbb9ff11402f2"}
Jul 08 17:09:54.638669 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:54.638624    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd" podStartSLOduration=2.627763876 podStartE2EDuration="11.638609568s" totalImagesPullingTime="0s" totalInitContainerRuntime="9.010845692s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:09:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:09:54.635857106 +0000 UTC m=+2131.349218554" watchObservedRunningTime="2026-07-08 17:09:54.638609568 +0000 UTC m=+2131.351970994"
Jul 08 17:09:55.198512 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:09:55.198474    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"
Jul 08 17:10:04.198495 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:10:04.198450    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"
Jul 08 17:10:04.200379 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:10:04.200349    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd" podUID="bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 08 17:10:14.198129 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:10:14.198091    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd" podUID="bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 08 17:10:24.198482 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:10:24.198430    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd" podUID="bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 08 17:10:34.199270 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:10:34.199221    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd" podUID="bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 08 17:10:44.198592 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:10:44.198549    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd" podUID="bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 08 17:10:54.198178 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:10:54.198120    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd" podUID="bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 08 17:11:04.198778 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:11:04.198729    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd" podUID="bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 08 17:11:14.198149 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:11:14.198098    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd" podUID="bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 08 17:11:24.198326 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:11:24.198279    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd" podUID="bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 08 17:11:34.198771 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:11:34.198722    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd" podUID="bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 08 17:11:44.199102 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:11:44.199062    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd" podUID="bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": dial tcp 10.133.0.64:8000: connect: connection refused"
Jul 08 17:11:54.208852 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:11:54.208768    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"
Jul 08 17:11:54.216553 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:11:54.216528    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"
Jul 08 17:11:58.578340 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:11:58.578304    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"]
Jul 08 17:11:58.609498 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:11:58.609450    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:11:58.669551 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:11:58.669520    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:12:13.602653 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:13.602587    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd" podUID="bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a" containerName="main" containerID="cri-o://a8c23f447813033467af1361dcf397ae55a18dea788e0f84892dbb9ff11402f2" gracePeriod=45
Jul 08 17:12:14.299937 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.299911    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"
Jul 08 17:12:14.408087 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.408000    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-home\") pod \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\" (UID: \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\") "
Jul 08 17:12:14.408087 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.408045    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-model-cache\") pod \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\" (UID: \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\") "
Jul 08 17:12:14.408087 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.408066    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-tmp-dir\") pod \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\" (UID: \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\") "
Jul 08 17:12:14.408375 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.408154    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-dshm\") pod \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\" (UID: \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\") "
Jul 08 17:12:14.408375 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.408174    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ctwgj\" (UniqueName: \"kubernetes.io/projected/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-kube-api-access-ctwgj\") pod \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\" (UID: \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\") "
Jul 08 17:12:14.408375 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.408200    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-kserve-provision-location\") pod \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\" (UID: \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\") "
Jul 08 17:12:14.408375 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.408223    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-tls-certs\") pod \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\" (UID: \"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a\") "
Jul 08 17:12:14.408375 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.408340    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-model-cache" (OuterVolumeSpecName: "model-cache") pod "bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a" (UID: "bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:12:14.408635 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.408595    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:12:14.408847 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.408819    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-home" (OuterVolumeSpecName: "home") pod "bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a" (UID: "bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:12:14.410396 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.410367    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-dshm" (OuterVolumeSpecName: "dshm") pod "bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a" (UID: "bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:12:14.410396 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.410374    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-kube-api-access-ctwgj" (OuterVolumeSpecName: "kube-api-access-ctwgj") pod "bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a" (UID: "bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a"). InnerVolumeSpecName "kube-api-access-ctwgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:12:14.410637 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.410617    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a" (UID: "bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:12:14.421527 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.421501    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a" (UID: "bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:12:14.427638 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.427609    2565 generic.go:356] "Generic (PLEG): container finished" podID="bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a" containerID="a8c23f447813033467af1361dcf397ae55a18dea788e0f84892dbb9ff11402f2" exitCode=0
Jul 08 17:12:14.427940 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.427754    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"
Jul 08 17:12:14.427940 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.427798    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd" event={"ID":"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a","Type":"ContainerDied","Data":"a8c23f447813033467af1361dcf397ae55a18dea788e0f84892dbb9ff11402f2"}
Jul 08 17:12:14.427940 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.427882    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd" event={"ID":"bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a","Type":"ContainerDied","Data":"49f64f18f8cf983de2219a5016358719855dc96d3ae5d226340c79c162ee9671"}
Jul 08 17:12:14.427940 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.427912    2565 scope.go:117] "RemoveContainer" containerID="a8c23f447813033467af1361dcf397ae55a18dea788e0f84892dbb9ff11402f2"
Jul 08 17:12:14.438100 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.438083    2565 scope.go:117] "RemoveContainer" containerID="cf3d505357d8d59ead8438c1ce28cf8f2c5001f3c8524af10dd0d9d3031ee6ab"
Jul 08 17:12:14.448534 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.448516    2565 scope.go:117] "RemoveContainer" containerID="a8c23f447813033467af1361dcf397ae55a18dea788e0f84892dbb9ff11402f2"
Jul 08 17:12:14.448740 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:12:14.448719    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8c23f447813033467af1361dcf397ae55a18dea788e0f84892dbb9ff11402f2\": container with ID starting with a8c23f447813033467af1361dcf397ae55a18dea788e0f84892dbb9ff11402f2 not found: ID does not exist" containerID="a8c23f447813033467af1361dcf397ae55a18dea788e0f84892dbb9ff11402f2"
Jul 08 17:12:14.448814 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.448748    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8c23f447813033467af1361dcf397ae55a18dea788e0f84892dbb9ff11402f2"} err="failed to get container status \"a8c23f447813033467af1361dcf397ae55a18dea788e0f84892dbb9ff11402f2\": rpc error: code = NotFound desc = could not find container \"a8c23f447813033467af1361dcf397ae55a18dea788e0f84892dbb9ff11402f2\": container with ID starting with a8c23f447813033467af1361dcf397ae55a18dea788e0f84892dbb9ff11402f2 not found: ID does not exist"
Jul 08 17:12:14.448814 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.448774    2565 scope.go:117] "RemoveContainer" containerID="cf3d505357d8d59ead8438c1ce28cf8f2c5001f3c8524af10dd0d9d3031ee6ab"
Jul 08 17:12:14.449007 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:12:14.448986    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf3d505357d8d59ead8438c1ce28cf8f2c5001f3c8524af10dd0d9d3031ee6ab\": container with ID starting with cf3d505357d8d59ead8438c1ce28cf8f2c5001f3c8524af10dd0d9d3031ee6ab not found: ID does not exist" containerID="cf3d505357d8d59ead8438c1ce28cf8f2c5001f3c8524af10dd0d9d3031ee6ab"
Jul 08 17:12:14.449069 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.449014    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf3d505357d8d59ead8438c1ce28cf8f2c5001f3c8524af10dd0d9d3031ee6ab"} err="failed to get container status \"cf3d505357d8d59ead8438c1ce28cf8f2c5001f3c8524af10dd0d9d3031ee6ab\": rpc error: code = NotFound desc = could not find container \"cf3d505357d8d59ead8438c1ce28cf8f2c5001f3c8524af10dd0d9d3031ee6ab\": container with ID starting with cf3d505357d8d59ead8438c1ce28cf8f2c5001f3c8524af10dd0d9d3031ee6ab not found: ID does not exist"
Jul 08 17:12:14.481633 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.481587    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a" (UID: "bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:12:14.509644 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.509615    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:12:14.509644 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.509642    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctwgj\" (UniqueName: \"kubernetes.io/projected/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-kube-api-access-ctwgj\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:12:14.509800 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.509653    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-kserve-provision-location\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:12:14.509800 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.509663    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:12:14.509800 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.509686    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:12:14.509800 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.509696    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:12:14.752965 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.752937    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"]
Jul 08 17:12:14.757314 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:14.757289    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd"]
Jul 08 17:12:15.210759 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:15.210717    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/router-with-refs-test-kserve-697958d7-hhdqd" podUID="bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 08 17:12:15.827274 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:15.827240    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a" path="/var/lib/kubelet/pods/bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a/volumes"
Jul 08 17:12:23.240379 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.240347    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"]
Jul 08 17:12:23.240938 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.240755    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a" containerName="storage-initializer"
Jul 08 17:12:23.240938 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.240768    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a" containerName="storage-initializer"
Jul 08 17:12:23.240938 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.240778    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a" containerName="main"
Jul 08 17:12:23.240938 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.240783    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a" containerName="main"
Jul 08 17:12:23.240938 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.240852    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc3ac9f3-d750-4cd9-a44e-32df8afd5e9a" containerName="main"
Jul 08 17:12:23.245925 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.245904    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:12:23.248798 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.248774    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 17:12:23.248979 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.248841    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-dockercfg-dzp8k\""
Jul 08 17:12:23.249071 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.248841    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv44d181485fad85e662eb092f3749502f-kserve-self-signed-certs\""
Jul 08 17:12:23.272366 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.272327    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"]
Jul 08 17:12:23.274921 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.274897    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"]
Jul 08 17:12:23.278451 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.278436    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"
Jul 08 17:12:23.321555 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.321523    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:12:23.333332 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.333308    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"]
Jul 08 17:12:23.400440 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.400406    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vgqc\" (UniqueName: \"kubernetes.io/projected/1ebffd2a-ed17-4633-8c22-e928a12f9286-kube-api-access-4vgqc\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd\" (UID: \"1ebffd2a-ed17-4633-8c22-e928a12f9286\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"
Jul 08 17:12:23.400605 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.400446    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99\" (UID: \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:12:23.400605 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.400474    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ebffd2a-ed17-4633-8c22-e928a12f9286-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd\" (UID: \"1ebffd2a-ed17-4633-8c22-e928a12f9286\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"
Jul 08 17:12:23.400605 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.400497    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd\" (UID: \"1ebffd2a-ed17-4633-8c22-e928a12f9286\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"
Jul 08 17:12:23.400605 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.400516    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd\" (UID: \"1ebffd2a-ed17-4633-8c22-e928a12f9286\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"
Jul 08 17:12:23.400605 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.400547    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjftb\" (UniqueName: \"kubernetes.io/projected/dce4cf96-d48d-482a-9184-8e2bc1ac3932-kube-api-access-rjftb\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99\" (UID: \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:12:23.400605 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.400602    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd\" (UID: \"1ebffd2a-ed17-4633-8c22-e928a12f9286\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"
Jul 08 17:12:23.400902 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.400622    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99\" (UID: \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:12:23.400902 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.400644    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd\" (UID: \"1ebffd2a-ed17-4633-8c22-e928a12f9286\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"
Jul 08 17:12:23.400902 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.400697    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99\" (UID: \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:12:23.400902 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.400715    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd\" (UID: \"1ebffd2a-ed17-4633-8c22-e928a12f9286\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"
Jul 08 17:12:23.400902 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.400734    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99\" (UID: \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:12:23.400902 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.400753    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99\" (UID: \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:12:23.400902 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.400831    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dce4cf96-d48d-482a-9184-8e2bc1ac3932-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99\" (UID: \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:12:23.501568 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.501480    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ebffd2a-ed17-4633-8c22-e928a12f9286-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd\" (UID: \"1ebffd2a-ed17-4633-8c22-e928a12f9286\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"
Jul 08 17:12:23.501568 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.501516    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd\" (UID: \"1ebffd2a-ed17-4633-8c22-e928a12f9286\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"
Jul 08 17:12:23.501568 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.501534    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd\" (UID: \"1ebffd2a-ed17-4633-8c22-e928a12f9286\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"
Jul 08 17:12:23.501892 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.501659    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjftb\" (UniqueName: \"kubernetes.io/projected/dce4cf96-d48d-482a-9184-8e2bc1ac3932-kube-api-access-rjftb\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99\" (UID: \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:12:23.501892 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.501801    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd\" (UID: \"1ebffd2a-ed17-4633-8c22-e928a12f9286\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"
Jul 08 17:12:23.501892 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.501837    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99\" (UID: \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:12:23.501892 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.501871    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd\" (UID: \"1ebffd2a-ed17-4633-8c22-e928a12f9286\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"
Jul 08 17:12:23.502112 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.501918    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99\" (UID: \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:12:23.502112 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.501950    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd\" (UID: \"1ebffd2a-ed17-4633-8c22-e928a12f9286\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"
Jul 08 17:12:23.502112 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.501990    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99\" (UID: \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:12:23.502112 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.501994    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd\" (UID: \"1ebffd2a-ed17-4633-8c22-e928a12f9286\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"
Jul 08 17:12:23.502112 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.502022    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99\" (UID: \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:12:23.502112 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.502057    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dce4cf96-d48d-482a-9184-8e2bc1ac3932-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99\" (UID: \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:12:23.502112 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.502092    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vgqc\" (UniqueName: \"kubernetes.io/projected/1ebffd2a-ed17-4633-8c22-e928a12f9286-kube-api-access-4vgqc\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd\" (UID: \"1ebffd2a-ed17-4633-8c22-e928a12f9286\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"
Jul 08 17:12:23.502574 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.502131    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99\" (UID: \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:12:23.502574 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.502155    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd\" (UID: \"1ebffd2a-ed17-4633-8c22-e928a12f9286\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"
Jul 08 17:12:23.502574 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.502174    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd\" (UID: \"1ebffd2a-ed17-4633-8c22-e928a12f9286\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"
Jul 08 17:12:23.502574 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.502244    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-home\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99\" (UID: \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:12:23.502574 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.502266    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99\" (UID: \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:12:23.502574 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.502434    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd\" (UID: \"1ebffd2a-ed17-4633-8c22-e928a12f9286\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"
Jul 08 17:12:23.502951 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.502593    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99\" (UID: \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:12:23.502951 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.502666    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99\" (UID: \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:12:23.504191 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.504161    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd\" (UID: \"1ebffd2a-ed17-4633-8c22-e928a12f9286\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"
Jul 08 17:12:23.504542 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.504510    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-dshm\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99\" (UID: \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:12:23.504664 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.504647    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dce4cf96-d48d-482a-9184-8e2bc1ac3932-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99\" (UID: \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:12:23.504774 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.504755    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ebffd2a-ed17-4633-8c22-e928a12f9286-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd\" (UID: \"1ebffd2a-ed17-4633-8c22-e928a12f9286\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"
Jul 08 17:12:23.514938 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.514917    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vgqc\" (UniqueName: \"kubernetes.io/projected/1ebffd2a-ed17-4633-8c22-e928a12f9286-kube-api-access-4vgqc\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd\" (UID: \"1ebffd2a-ed17-4633-8c22-e928a12f9286\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"
Jul 08 17:12:23.515049 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.515026    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjftb\" (UniqueName: \"kubernetes.io/projected/dce4cf96-d48d-482a-9184-8e2bc1ac3932-kube-api-access-rjftb\") pod \"llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99\" (UID: \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:12:23.557054 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.557022    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:12:23.588550 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.588522    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"
Jul 08 17:12:23.727074 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.727045    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"]
Jul 08 17:12:23.728476 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:12:23.728445    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce4cf96_d48d_482a_9184_8e2bc1ac3932.slice/crio-6d5510a01c79682512662401884b1e841f36d7613be199035d0c46d76c19be5a WatchSource:0}: Error finding container 6d5510a01c79682512662401884b1e841f36d7613be199035d0c46d76c19be5a: Status 404 returned error can't find the container with id 6d5510a01c79682512662401884b1e841f36d7613be199035d0c46d76c19be5a
Jul 08 17:12:23.749347 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.749316    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"]
Jul 08 17:12:23.750168 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:12:23.750141    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ebffd2a_ed17_4633_8c22_e928a12f9286.slice/crio-69b0522c08c840d0da0ee8700203bf8bb027e216cdc546b8a17c46afd00674ca WatchSource:0}: Error finding container 69b0522c08c840d0da0ee8700203bf8bb027e216cdc546b8a17c46afd00674ca: Status 404 returned error can't find the container with id 69b0522c08c840d0da0ee8700203bf8bb027e216cdc546b8a17c46afd00674ca
Jul 08 17:12:23.754640 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:23.754619    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 17:12:24.497492 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:24.497441    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd" event={"ID":"1ebffd2a-ed17-4633-8c22-e928a12f9286","Type":"ContainerStarted","Data":"029ce918d05eff69bca384f7e39e0033ada8fab19f86cdf66046f83cd9766ba4"}
Jul 08 17:12:24.498003 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:24.497543    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd" event={"ID":"1ebffd2a-ed17-4633-8c22-e928a12f9286","Type":"ContainerStarted","Data":"69b0522c08c840d0da0ee8700203bf8bb027e216cdc546b8a17c46afd00674ca"}
Jul 08 17:12:24.504131 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:24.504091    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99" event={"ID":"dce4cf96-d48d-482a-9184-8e2bc1ac3932","Type":"ContainerStarted","Data":"b360e6657c7fee032a86e9b397b411d7e38fc4027040c4b65d60615a9433fb51"}
Jul 08 17:12:24.504289 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:24.504173    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:12:24.504289 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:24.504194    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99" event={"ID":"dce4cf96-d48d-482a-9184-8e2bc1ac3932","Type":"ContainerStarted","Data":"6d5510a01c79682512662401884b1e841f36d7613be199035d0c46d76c19be5a"}
Jul 08 17:12:25.409093 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:25.409049    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:12:25.511938 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:25.511904    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99" event={"ID":"dce4cf96-d48d-482a-9184-8e2bc1ac3932","Type":"ContainerStarted","Data":"b486e800825c4e3a801c41a91a13badf8cd8d4f16a68d783d7de7ca1dd7cdf0a"}
Jul 08 17:12:30.437787 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:30.437668    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:12:30.556213 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:30.556171    2565 generic.go:356] "Generic (PLEG): container finished" podID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerID="b486e800825c4e3a801c41a91a13badf8cd8d4f16a68d783d7de7ca1dd7cdf0a" exitCode=0
Jul 08 17:12:30.556377 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:30.556239    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99" event={"ID":"dce4cf96-d48d-482a-9184-8e2bc1ac3932","Type":"ContainerDied","Data":"b486e800825c4e3a801c41a91a13badf8cd8d4f16a68d783d7de7ca1dd7cdf0a"}
Jul 08 17:12:31.569465 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:31.569426    2565 generic.go:356] "Generic (PLEG): container finished" podID="1ebffd2a-ed17-4633-8c22-e928a12f9286" containerID="029ce918d05eff69bca384f7e39e0033ada8fab19f86cdf66046f83cd9766ba4" exitCode=0
Jul 08 17:12:31.570166 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:31.569521    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd" event={"ID":"1ebffd2a-ed17-4633-8c22-e928a12f9286","Type":"ContainerDied","Data":"029ce918d05eff69bca384f7e39e0033ada8fab19f86cdf66046f83cd9766ba4"}
Jul 08 17:12:31.575224 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:31.574803    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99" event={"ID":"dce4cf96-d48d-482a-9184-8e2bc1ac3932","Type":"ContainerStarted","Data":"d0e59f7415b95003e64c32b344f387d82e655a554d48c76060e29239ea79e3c4"}
Jul 08 17:12:31.630646 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:31.630588    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99" podStartSLOduration=2.691881718 podStartE2EDuration="8.630565691s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.938683973s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:12:31.625900747 +0000 UTC m=+2288.339262203" watchObservedRunningTime="2026-07-08 17:12:31.630565691 +0000 UTC m=+2288.343927121"
Jul 08 17:12:32.582289 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:32.582251    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd" event={"ID":"1ebffd2a-ed17-4633-8c22-e928a12f9286","Type":"ContainerStarted","Data":"03d08a3a1ba64da0ce2b70bdb8bb18d18437dfb38e69599c0f4ad649aa65f603"}
Jul 08 17:12:32.607549 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:32.607483    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd" podStartSLOduration=2.072413573 podStartE2EDuration="9.607461786s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.535048213s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:12:32.606335223 +0000 UTC m=+2289.319696691" watchObservedRunningTime="2026-07-08 17:12:32.607461786 +0000 UTC m=+2289.320823214"
Jul 08 17:12:33.557327 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:33.557288    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:12:33.557517 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:33.557348    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:12:33.558844 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:33.558812    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99" podUID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8001/health\": dial tcp 10.133.0.65:8001: connect: connection refused"
Jul 08 17:12:33.572990 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:33.572958    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:12:33.589406 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:33.589376    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"
Jul 08 17:12:33.589868 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:33.589433    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"
Jul 08 17:12:33.591046 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:33.591019    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd" podUID="1ebffd2a-ed17-4633-8c22-e928a12f9286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 08 17:12:33.913007 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:33.912712    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:12:33.932916 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:33.932887    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 08 17:12:34.028638 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:34.028596    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:12:43.557693 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:43.557611    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99" podUID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8001/health\": dial tcp 10.133.0.65:8001: connect: connection refused"
Jul 08 17:12:43.589381 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:43.589340    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd" podUID="1ebffd2a-ed17-4633-8c22-e928a12f9286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 08 17:12:48.969919 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:48.969862    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" containerName="main" containerID="cri-o://07ff61abcb486c810e8a209de4fe8a61c14bd022d34846bafb2e75eacedd5840" gracePeriod=45
Jul 08 17:12:49.428658 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:49.428609    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" podUID="52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": dial tcp 10.133.0.62:8000: connect: connection refused"
Jul 08 17:12:49.702453 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:49.702344    2565 generic.go:356] "Generic (PLEG): container finished" podID="52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" containerID="07ff61abcb486c810e8a209de4fe8a61c14bd022d34846bafb2e75eacedd5840" exitCode=0
Jul 08 17:12:49.702453 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:49.702426    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1","Type":"ContainerDied","Data":"07ff61abcb486c810e8a209de4fe8a61c14bd022d34846bafb2e75eacedd5840"}
Jul 08 17:12:49.770996 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:49.770972    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:12:49.887833 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:49.887792    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-dshm\") pod \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") "
Jul 08 17:12:49.887833 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:49.887841    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-home\") pod \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") "
Jul 08 17:12:49.888096 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:49.887892    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-tls-certs\") pod \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") "
Jul 08 17:12:49.888096 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:49.887917    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-model-cache\") pod \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") "
Jul 08 17:12:49.888096 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:49.887971    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8k9vj\" (UniqueName: \"kubernetes.io/projected/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-kube-api-access-8k9vj\") pod \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") "
Jul 08 17:12:49.888096 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:49.887995    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-tmp-dir\") pod \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") "
Jul 08 17:12:49.888300 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:49.888174    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\" (UID: \"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1\") "
Jul 08 17:12:49.888300 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:49.888253    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-model-cache" (OuterVolumeSpecName: "model-cache") pod "52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" (UID: "52d4f4f3-2d00-4f5a-ba45-d4e44f350af1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:12:49.888810 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:49.888774    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-home" (OuterVolumeSpecName: "home") pod "52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" (UID: "52d4f4f3-2d00-4f5a-ba45-d4e44f350af1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:12:49.888948 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:49.888936    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:12:49.891174 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:49.891147    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" (UID: "52d4f4f3-2d00-4f5a-ba45-d4e44f350af1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:12:49.891280 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:49.891151    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-kube-api-access-8k9vj" (OuterVolumeSpecName: "kube-api-access-8k9vj") pod "52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" (UID: "52d4f4f3-2d00-4f5a-ba45-d4e44f350af1"). InnerVolumeSpecName "kube-api-access-8k9vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:12:49.891333 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:49.891295    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-dshm" (OuterVolumeSpecName: "dshm") pod "52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" (UID: "52d4f4f3-2d00-4f5a-ba45-d4e44f350af1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:12:49.891826 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:49.891805    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75" (OuterVolumeSpecName: "kserve-pvc-source") pod "52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" (UID: "52d4f4f3-2d00-4f5a-ba45-d4e44f350af1"). InnerVolumeSpecName "pvc-6394c2ef-5d09-4573-b34c-dae5966e5434". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 08 17:12:49.906185 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:49.906152    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" (UID: "52d4f4f3-2d00-4f5a-ba45-d4e44f350af1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:12:49.989892 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:49.989803    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:12:49.989892 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:49.989846    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k9vj\" (UniqueName: \"kubernetes.io/projected/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-kube-api-access-8k9vj\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:12:49.989892 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:49.989863    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:12:49.989892 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:49.989879    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:12:49.989892 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:49.989893    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:12:50.713305 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:50.713261    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0" event={"ID":"52d4f4f3-2d00-4f5a-ba45-d4e44f350af1","Type":"ContainerDied","Data":"049841a98a3b829ba7f62fe51dd5ed5258cf90c718dd51c99172c9a04b7a51b2"}
Jul 08 17:12:50.713515 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:50.713338    2565 scope.go:117] "RemoveContainer" containerID="07ff61abcb486c810e8a209de4fe8a61c14bd022d34846bafb2e75eacedd5840"
Jul 08 17:12:50.713515 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:50.713497    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"
Jul 08 17:12:50.745246 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:50.745210    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 08 17:12:50.762941 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:50.762911    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0"]
Jul 08 17:12:50.774286 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:50.774254    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 08 17:12:51.823353 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:51.823313    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" path="/var/lib/kubelet/pods/52d4f4f3-2d00-4f5a-ba45-d4e44f350af1/volumes"
Jul 08 17:12:53.557720 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:53.557658    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99" podUID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8001/health\": dial tcp 10.133.0.65:8001: connect: connection refused"
Jul 08 17:12:53.589601 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:12:53.589566    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd" podUID="1ebffd2a-ed17-4633-8c22-e928a12f9286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 08 17:13:03.557529 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:03.557473    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99" podUID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8001/health\": dial tcp 10.133.0.65:8001: connect: connection refused"
Jul 08 17:13:03.589384 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:03.589339    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd" podUID="1ebffd2a-ed17-4633-8c22-e928a12f9286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 08 17:13:05.798298 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:05.798221    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" podUID="0e7a242b-c2d7-49ae-8c07-eb8743793960" containerName="main" containerID="cri-o://2bcc12d7127e510c6c1d51753d44897665410472499901d94cb074d075d116e9" gracePeriod=45
Jul 08 17:13:06.562626 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.562587    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 17:13:06.650181 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.650091    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e7a242b-c2d7-49ae-8c07-eb8743793960-model-cache\") pod \"0e7a242b-c2d7-49ae-8c07-eb8743793960\" (UID: \"0e7a242b-c2d7-49ae-8c07-eb8743793960\") "
Jul 08 17:13:06.650181 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.650135    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e7a242b-c2d7-49ae-8c07-eb8743793960-tls-certs\") pod \"0e7a242b-c2d7-49ae-8c07-eb8743793960\" (UID: \"0e7a242b-c2d7-49ae-8c07-eb8743793960\") "
Jul 08 17:13:06.650434 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.650227    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-pvc-source\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") pod \"0e7a242b-c2d7-49ae-8c07-eb8743793960\" (UID: \"0e7a242b-c2d7-49ae-8c07-eb8743793960\") "
Jul 08 17:13:06.650434 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.650265    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e7a242b-c2d7-49ae-8c07-eb8743793960-dshm\") pod \"0e7a242b-c2d7-49ae-8c07-eb8743793960\" (UID: \"0e7a242b-c2d7-49ae-8c07-eb8743793960\") "
Jul 08 17:13:06.650434 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.650283    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e7a242b-c2d7-49ae-8c07-eb8743793960-tmp-dir\") pod \"0e7a242b-c2d7-49ae-8c07-eb8743793960\" (UID: \"0e7a242b-c2d7-49ae-8c07-eb8743793960\") "
Jul 08 17:13:06.650434 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.650344    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7a242b-c2d7-49ae-8c07-eb8743793960-model-cache" (OuterVolumeSpecName: "model-cache") pod "0e7a242b-c2d7-49ae-8c07-eb8743793960" (UID: "0e7a242b-c2d7-49ae-8c07-eb8743793960"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:13:06.650434 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.650386    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e7a242b-c2d7-49ae-8c07-eb8743793960-home\") pod \"0e7a242b-c2d7-49ae-8c07-eb8743793960\" (UID: \"0e7a242b-c2d7-49ae-8c07-eb8743793960\") "
Jul 08 17:13:06.650762 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.650450    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r99gf\" (UniqueName: \"kubernetes.io/projected/0e7a242b-c2d7-49ae-8c07-eb8743793960-kube-api-access-r99gf\") pod \"0e7a242b-c2d7-49ae-8c07-eb8743793960\" (UID: \"0e7a242b-c2d7-49ae-8c07-eb8743793960\") "
Jul 08 17:13:06.650824 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.650795    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0e7a242b-c2d7-49ae-8c07-eb8743793960-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:13:06.651047 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.651003    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7a242b-c2d7-49ae-8c07-eb8743793960-home" (OuterVolumeSpecName: "home") pod "0e7a242b-c2d7-49ae-8c07-eb8743793960" (UID: "0e7a242b-c2d7-49ae-8c07-eb8743793960"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:13:06.653141 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.653114    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e7a242b-c2d7-49ae-8c07-eb8743793960-kube-api-access-r99gf" (OuterVolumeSpecName: "kube-api-access-r99gf") pod "0e7a242b-c2d7-49ae-8c07-eb8743793960" (UID: "0e7a242b-c2d7-49ae-8c07-eb8743793960"). InnerVolumeSpecName "kube-api-access-r99gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:13:06.653839 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.653814    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7a242b-c2d7-49ae-8c07-eb8743793960-dshm" (OuterVolumeSpecName: "dshm") pod "0e7a242b-c2d7-49ae-8c07-eb8743793960" (UID: "0e7a242b-c2d7-49ae-8c07-eb8743793960"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:13:06.653917 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.653829    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e7a242b-c2d7-49ae-8c07-eb8743793960-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0e7a242b-c2d7-49ae-8c07-eb8743793960" (UID: "0e7a242b-c2d7-49ae-8c07-eb8743793960"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:13:06.654321 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.654278    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75" (OuterVolumeSpecName: "kserve-pvc-source") pod "0e7a242b-c2d7-49ae-8c07-eb8743793960" (UID: "0e7a242b-c2d7-49ae-8c07-eb8743793960"). InnerVolumeSpecName "pvc-6394c2ef-5d09-4573-b34c-dae5966e5434". PluginName "kubernetes.io/csi", VolumeGIDValue ""
Jul 08 17:13:06.662836 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.662811    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e7a242b-c2d7-49ae-8c07-eb8743793960-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0e7a242b-c2d7-49ae-8c07-eb8743793960" (UID: "0e7a242b-c2d7-49ae-8c07-eb8743793960"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:13:06.752015 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.751975    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r99gf\" (UniqueName: \"kubernetes.io/projected/0e7a242b-c2d7-49ae-8c07-eb8743793960-kube-api-access-r99gf\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:13:06.752015 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.752010    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0e7a242b-c2d7-49ae-8c07-eb8743793960-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:13:06.752230 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.752041    2565 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") on node \"ip-10-0-128-142.ec2.internal\" "
Jul 08 17:13:06.752230 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.752050    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0e7a242b-c2d7-49ae-8c07-eb8743793960-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:13:06.752230 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.752060    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0e7a242b-c2d7-49ae-8c07-eb8743793960-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:13:06.752230 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.752069    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0e7a242b-c2d7-49ae-8c07-eb8743793960-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:13:06.780689 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.780656    2565 operation_generator.go:895] UnmountDevice succeeded for volume "pvc-6394c2ef-5d09-4573-b34c-dae5966e5434" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75") on node "ip-10-0-128-142.ec2.internal"
Jul 08 17:13:06.817057 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.817029    2565 generic.go:356] "Generic (PLEG): container finished" podID="0e7a242b-c2d7-49ae-8c07-eb8743793960" containerID="2bcc12d7127e510c6c1d51753d44897665410472499901d94cb074d075d116e9" exitCode=0
Jul 08 17:13:06.817437 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.817078    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"
Jul 08 17:13:06.817437 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.817084    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"0e7a242b-c2d7-49ae-8c07-eb8743793960","Type":"ContainerDied","Data":"2bcc12d7127e510c6c1d51753d44897665410472499901d94cb074d075d116e9"}
Jul 08 17:13:06.817437 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.817204    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1" event={"ID":"0e7a242b-c2d7-49ae-8c07-eb8743793960","Type":"ContainerDied","Data":"545d2e0871d179d8a0d8531d3795b648d626181ec0bb336a03770f9cf3609974"}
Jul 08 17:13:06.817437 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.817226    2565 scope.go:117] "RemoveContainer" containerID="2bcc12d7127e510c6c1d51753d44897665410472499901d94cb074d075d116e9"
Jul 08 17:13:06.827319 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.827302    2565 scope.go:117] "RemoveContainer" containerID="2bcc12d7127e510c6c1d51753d44897665410472499901d94cb074d075d116e9"
Jul 08 17:13:06.827713 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:13:06.827608    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bcc12d7127e510c6c1d51753d44897665410472499901d94cb074d075d116e9\": container with ID starting with 2bcc12d7127e510c6c1d51753d44897665410472499901d94cb074d075d116e9 not found: ID does not exist" containerID="2bcc12d7127e510c6c1d51753d44897665410472499901d94cb074d075d116e9"
Jul 08 17:13:06.827713 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.827641    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bcc12d7127e510c6c1d51753d44897665410472499901d94cb074d075d116e9"} err="failed to get container status \"2bcc12d7127e510c6c1d51753d44897665410472499901d94cb074d075d116e9\": rpc error: code = NotFound desc = could not find container \"2bcc12d7127e510c6c1d51753d44897665410472499901d94cb074d075d116e9\": container with ID starting with 2bcc12d7127e510c6c1d51753d44897665410472499901d94cb074d075d116e9 not found: ID does not exist"
Jul 08 17:13:06.845731 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.845703    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 08 17:13:06.851143 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.851115    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-pvc-router-manage-2577e794-kserve-mn-0-1"]
Jul 08 17:13:06.853410 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:06.853385    2565 reconciler_common.go:299] "Volume detached for volume \"pvc-6394c2ef-5d09-4573-b34c-dae5966e5434\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e360b4cc8b13cd75\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"csi-2dacefae46c7a149c270f1437d3b1a0444209eeb1f65f5a21ee1f03bbc64e3c5\""
Jul 08 17:13:07.825007 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:07.824967    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e7a242b-c2d7-49ae-8c07-eb8743793960" path="/var/lib/kubelet/pods/0e7a242b-c2d7-49ae-8c07-eb8743793960/volumes"
Jul 08 17:13:11.675220 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.675184    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"]
Jul 08 17:13:11.675900 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.675879    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e7a242b-c2d7-49ae-8c07-eb8743793960" containerName="main"
Jul 08 17:13:11.675969 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.675905    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e7a242b-c2d7-49ae-8c07-eb8743793960" containerName="main"
Jul 08 17:13:11.675969 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.675920    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" containerName="main"
Jul 08 17:13:11.675969 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.675928    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" containerName="main"
Jul 08 17:13:11.676069 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.676017    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e7a242b-c2d7-49ae-8c07-eb8743793960" containerName="main"
Jul 08 17:13:11.676069 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.676029    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="52d4f4f3-2d00-4f5a-ba45-d4e44f350af1" containerName="main"
Jul 08 17:13:11.679629 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.679610    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"
Jul 08 17:13:11.682892 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.682867    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 08 17:13:11.695569 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.695538    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"]
Jul 08 17:13:11.701927 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.701890    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:13:11.803208 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.803133    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-tmp-dir\") pod \"stop-feature-test-kserve-7c7845f79f-knxs6\" (UID: \"32cee549-134c-4fad-8b17-ed32953bde74\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"
Jul 08 17:13:11.803425 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.803254    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32cee549-134c-4fad-8b17-ed32953bde74-tls-certs\") pod \"stop-feature-test-kserve-7c7845f79f-knxs6\" (UID: \"32cee549-134c-4fad-8b17-ed32953bde74\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"
Jul 08 17:13:11.803425 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.803287    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-home\") pod \"stop-feature-test-kserve-7c7845f79f-knxs6\" (UID: \"32cee549-134c-4fad-8b17-ed32953bde74\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"
Jul 08 17:13:11.803425 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.803309    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-kserve-provision-location\") pod \"stop-feature-test-kserve-7c7845f79f-knxs6\" (UID: \"32cee549-134c-4fad-8b17-ed32953bde74\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"
Jul 08 17:13:11.803620 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.803415    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-dshm\") pod \"stop-feature-test-kserve-7c7845f79f-knxs6\" (UID: \"32cee549-134c-4fad-8b17-ed32953bde74\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"
Jul 08 17:13:11.803620 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.803477    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-model-cache\") pod \"stop-feature-test-kserve-7c7845f79f-knxs6\" (UID: \"32cee549-134c-4fad-8b17-ed32953bde74\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"
Jul 08 17:13:11.803620 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.803516    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9tq9\" (UniqueName: \"kubernetes.io/projected/32cee549-134c-4fad-8b17-ed32953bde74-kube-api-access-t9tq9\") pod \"stop-feature-test-kserve-7c7845f79f-knxs6\" (UID: \"32cee549-134c-4fad-8b17-ed32953bde74\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"
Jul 08 17:13:11.904834 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.904793    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-tmp-dir\") pod \"stop-feature-test-kserve-7c7845f79f-knxs6\" (UID: \"32cee549-134c-4fad-8b17-ed32953bde74\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"
Jul 08 17:13:11.905033 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.904896    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32cee549-134c-4fad-8b17-ed32953bde74-tls-certs\") pod \"stop-feature-test-kserve-7c7845f79f-knxs6\" (UID: \"32cee549-134c-4fad-8b17-ed32953bde74\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"
Jul 08 17:13:11.905033 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.904932    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-home\") pod \"stop-feature-test-kserve-7c7845f79f-knxs6\" (UID: \"32cee549-134c-4fad-8b17-ed32953bde74\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"
Jul 08 17:13:11.905033 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.904962    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-kserve-provision-location\") pod \"stop-feature-test-kserve-7c7845f79f-knxs6\" (UID: \"32cee549-134c-4fad-8b17-ed32953bde74\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"
Jul 08 17:13:11.905220 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.905035    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-dshm\") pod \"stop-feature-test-kserve-7c7845f79f-knxs6\" (UID: \"32cee549-134c-4fad-8b17-ed32953bde74\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"
Jul 08 17:13:11.905220 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.905103    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-model-cache\") pod \"stop-feature-test-kserve-7c7845f79f-knxs6\" (UID: \"32cee549-134c-4fad-8b17-ed32953bde74\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"
Jul 08 17:13:11.905220 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.905143    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9tq9\" (UniqueName: \"kubernetes.io/projected/32cee549-134c-4fad-8b17-ed32953bde74-kube-api-access-t9tq9\") pod \"stop-feature-test-kserve-7c7845f79f-knxs6\" (UID: \"32cee549-134c-4fad-8b17-ed32953bde74\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"
Jul 08 17:13:11.905367 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.905262    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-tmp-dir\") pod \"stop-feature-test-kserve-7c7845f79f-knxs6\" (UID: \"32cee549-134c-4fad-8b17-ed32953bde74\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"
Jul 08 17:13:11.905441 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.905417    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-kserve-provision-location\") pod \"stop-feature-test-kserve-7c7845f79f-knxs6\" (UID: \"32cee549-134c-4fad-8b17-ed32953bde74\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"
Jul 08 17:13:11.905495 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.905467    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-model-cache\") pod \"stop-feature-test-kserve-7c7845f79f-knxs6\" (UID: \"32cee549-134c-4fad-8b17-ed32953bde74\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"
Jul 08 17:13:11.905728 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.905700    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-home\") pod \"stop-feature-test-kserve-7c7845f79f-knxs6\" (UID: \"32cee549-134c-4fad-8b17-ed32953bde74\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"
Jul 08 17:13:11.908184 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.908157    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32cee549-134c-4fad-8b17-ed32953bde74-tls-certs\") pod \"stop-feature-test-kserve-7c7845f79f-knxs6\" (UID: \"32cee549-134c-4fad-8b17-ed32953bde74\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"
Jul 08 17:13:11.908568 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.908541    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-dshm\") pod \"stop-feature-test-kserve-7c7845f79f-knxs6\" (UID: \"32cee549-134c-4fad-8b17-ed32953bde74\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"
Jul 08 17:13:11.919097 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.919059    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9tq9\" (UniqueName: \"kubernetes.io/projected/32cee549-134c-4fad-8b17-ed32953bde74-kube-api-access-t9tq9\") pod \"stop-feature-test-kserve-7c7845f79f-knxs6\" (UID: \"32cee549-134c-4fad-8b17-ed32953bde74\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"
Jul 08 17:13:11.983353 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.983258    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:13:11.992300 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:11.992262    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"
Jul 08 17:13:12.172853 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:12.172818    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"]
Jul 08 17:13:12.175782 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:13:12.175754    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32cee549_134c_4fad_8b17_ed32953bde74.slice/crio-6332786b3d1ae2debcecc282e507283042b64782384c4a2e1f9d57ef68beb193 WatchSource:0}: Error finding container 6332786b3d1ae2debcecc282e507283042b64782384c4a2e1f9d57ef68beb193: Status 404 returned error can't find the container with id 6332786b3d1ae2debcecc282e507283042b64782384c4a2e1f9d57ef68beb193
Jul 08 17:13:12.858512 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:12.858473    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6" event={"ID":"32cee549-134c-4fad-8b17-ed32953bde74","Type":"ContainerStarted","Data":"dadad4379ae59be752be7ea4886e14502af9433c3493aa27851dae3c3144aa2b"}
Jul 08 17:13:12.858891 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:12.858539    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6" event={"ID":"32cee549-134c-4fad-8b17-ed32953bde74","Type":"ContainerStarted","Data":"6332786b3d1ae2debcecc282e507283042b64782384c4a2e1f9d57ef68beb193"}
Jul 08 17:13:13.558630 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:13.558221    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99" podUID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8001/health\": dial tcp 10.133.0.65:8001: connect: connection refused"
Jul 08 17:13:13.589652 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:13.589605    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd" podUID="1ebffd2a-ed17-4633-8c22-e928a12f9286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 08 17:13:18.904389 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:18.904343    2565 generic.go:356] "Generic (PLEG): container finished" podID="32cee549-134c-4fad-8b17-ed32953bde74" containerID="dadad4379ae59be752be7ea4886e14502af9433c3493aa27851dae3c3144aa2b" exitCode=0
Jul 08 17:13:18.904900 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:18.904413    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6" event={"ID":"32cee549-134c-4fad-8b17-ed32953bde74","Type":"ContainerDied","Data":"dadad4379ae59be752be7ea4886e14502af9433c3493aa27851dae3c3144aa2b"}
Jul 08 17:13:19.915188 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:19.915149    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6" event={"ID":"32cee549-134c-4fad-8b17-ed32953bde74","Type":"ContainerStarted","Data":"cbb7c8445c0c9b7f5538083d5fa3a151037b45e1f2715e3d6054a24c174bb040"}
Jul 08 17:13:19.944079 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:19.944009    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6" podStartSLOduration=3.019240209 podStartE2EDuration="8.943984664s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.924744455s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:13:19.94196384 +0000 UTC m=+2336.655325305" watchObservedRunningTime="2026-07-08 17:13:19.943984664 +0000 UTC m=+2336.657346092"
Jul 08 17:13:19.993128 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:19.993094    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"
Jul 08 17:13:21.993026 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:21.992979    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"
Jul 08 17:13:21.995159 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:21.995128    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6" podUID="32cee549-134c-4fad-8b17-ed32953bde74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 08 17:13:22.299970 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:22.299862    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:13:23.557939 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:23.557876    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99" podUID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8001/health\": dial tcp 10.133.0.65:8001: connect: connection refused"
Jul 08 17:13:23.589267 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:23.589220    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd" podUID="1ebffd2a-ed17-4633-8c22-e928a12f9286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 08 17:13:31.993601 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:31.993548    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6" podUID="32cee549-134c-4fad-8b17-ed32953bde74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 08 17:13:33.557912 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:33.557859    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99" podUID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8001/health\": dial tcp 10.133.0.65:8001: connect: connection refused"
Jul 08 17:13:33.589385 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:33.589337    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd" podUID="1ebffd2a-ed17-4633-8c22-e928a12f9286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 08 17:13:41.993043 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:41.992994    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6" podUID="32cee549-134c-4fad-8b17-ed32953bde74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 08 17:13:43.558238 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:43.558187    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99" podUID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8001/health\": dial tcp 10.133.0.65:8001: connect: connection refused"
Jul 08 17:13:43.589630 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:43.589578    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd" podUID="1ebffd2a-ed17-4633-8c22-e928a12f9286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 08 17:13:51.993654 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:51.993600    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6" podUID="32cee549-134c-4fad-8b17-ed32953bde74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 08 17:13:53.557529 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:53.557473    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99" podUID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8001/health\": dial tcp 10.133.0.65:8001: connect: connection refused"
Jul 08 17:13:53.589215 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:13:53.589165    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd" podUID="1ebffd2a-ed17-4633-8c22-e928a12f9286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 08 17:14:01.993194 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:14:01.993145    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6" podUID="32cee549-134c-4fad-8b17-ed32953bde74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 08 17:14:03.558202 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:14:03.558156    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99" podUID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8001/health\": dial tcp 10.133.0.65:8001: connect: connection refused"
Jul 08 17:14:03.589477 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:14:03.589441    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd" podUID="1ebffd2a-ed17-4633-8c22-e928a12f9286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 08 17:14:11.992957 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:14:11.992908    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6" podUID="32cee549-134c-4fad-8b17-ed32953bde74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 08 17:14:13.557904 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:14:13.557849    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99" podUID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8001/health\": dial tcp 10.133.0.65:8001: connect: connection refused"
Jul 08 17:14:13.588965 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:14:13.588923    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd" podUID="1ebffd2a-ed17-4633-8c22-e928a12f9286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 08 17:14:17.597138 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:14:17.597093    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:14:21.992970 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:14:21.992919    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6" podUID="32cee549-134c-4fad-8b17-ed32953bde74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 08 17:14:23.558914 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:14:23.558859    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99" podUID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8001/health\": dial tcp 10.133.0.65:8001: connect: connection refused"
Jul 08 17:14:23.589833 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:14:23.589787    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd" podUID="1ebffd2a-ed17-4633-8c22-e928a12f9286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 08 17:14:31.993841 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:14:31.993784    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6" podUID="32cee549-134c-4fad-8b17-ed32953bde74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 08 17:14:33.557585 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:14:33.557535    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99" podUID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8001/health\": dial tcp 10.133.0.65:8001: connect: connection refused"
Jul 08 17:14:33.589917 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:14:33.589874    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd" podUID="1ebffd2a-ed17-4633-8c22-e928a12f9286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 08 17:14:41.992882 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:14:41.992832    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6" podUID="32cee549-134c-4fad-8b17-ed32953bde74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 08 17:14:43.557738 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:14:43.557668    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99" podUID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8001/health\": dial tcp 10.133.0.65:8001: connect: connection refused"
Jul 08 17:14:43.589935 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:14:43.589904    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd" podUID="1ebffd2a-ed17-4633-8c22-e928a12f9286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 08 17:14:51.993607 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:14:51.993511    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6" podUID="32cee549-134c-4fad-8b17-ed32953bde74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 08 17:14:53.558019 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:14:53.557974    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99" podUID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8001/health\": dial tcp 10.133.0.65:8001: connect: connection refused"
Jul 08 17:14:53.589669 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:14:53.589634    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd" podUID="1ebffd2a-ed17-4633-8c22-e928a12f9286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 08 17:14:59.878111 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:14:59.878076    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:15:01.992900 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:15:01.992850    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6" podUID="32cee549-134c-4fad-8b17-ed32953bde74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 08 17:15:03.558314 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:15:03.558263    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99" podUID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8001/health\": dial tcp 10.133.0.65:8001: connect: connection refused"
Jul 08 17:15:03.589395 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:15:03.589357    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd" podUID="1ebffd2a-ed17-4633-8c22-e928a12f9286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 08 17:15:11.993513 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:15:11.993467    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6" podUID="32cee549-134c-4fad-8b17-ed32953bde74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 08 17:15:13.557716 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:15:13.557655    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99" podUID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8001/health\": dial tcp 10.133.0.65:8001: connect: connection refused"
Jul 08 17:15:13.589413 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:15:13.589380    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd" podUID="1ebffd2a-ed17-4633-8c22-e928a12f9286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 08 17:15:21.993337 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:15:21.993286    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6" podUID="32cee549-134c-4fad-8b17-ed32953bde74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 08 17:15:23.558030 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:15:23.557968    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99" podUID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerName="main" probeResult="failure" output="Get \"https://10.133.0.65:8001/health\": dial tcp 10.133.0.65:8001: connect: connection refused"
Jul 08 17:15:23.589118 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:15:23.589076    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd" podUID="1ebffd2a-ed17-4633-8c22-e928a12f9286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 08 17:15:31.992652 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:15:31.992606    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6" podUID="32cee549-134c-4fad-8b17-ed32953bde74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 08 17:15:33.568301 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:15:33.568264    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:15:33.583055 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:15:33.583028    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:15:33.598970 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:15:33.598946    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"
Jul 08 17:15:33.606581 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:15:33.606557    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"
Jul 08 17:15:41.992816 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:15:41.992772    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6" podUID="32cee549-134c-4fad-8b17-ed32953bde74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 08 17:15:48.392816 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:15:48.392779    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:15:48.395912 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:15:48.395885    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"]
Jul 08 17:15:48.400193 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:15:48.400169    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"]
Jul 08 17:15:48.454388 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:15:48.454338    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:15:51.992777 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:15:51.992737    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6" podUID="32cee549-134c-4fad-8b17-ed32953bde74" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": dial tcp 10.133.0.67:8000: connect: connection refused"
Jul 08 17:16:02.002872 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:02.002839    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"
Jul 08 17:16:02.010435 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:02.010412    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"
Jul 08 17:16:03.420234 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:03.420198    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99" podUID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerName="main" containerID="cri-o://d0e59f7415b95003e64c32b344f387d82e655a554d48c76060e29239ea79e3c4" gracePeriod=45
Jul 08 17:16:03.423425 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:03.423397    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd" podUID="1ebffd2a-ed17-4633-8c22-e928a12f9286" containerName="main" containerID="cri-o://03d08a3a1ba64da0ce2b70bdb8bb18d18437dfb38e69599c0f4ad649aa65f603" gracePeriod=45
Jul 08 17:16:03.450592 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:03.450558    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"]
Jul 08 17:16:03.480550 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:03.480507    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:16:03.589230 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:03.589193    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:16:03.600228 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:03.600193    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd" podUID="1ebffd2a-ed17-4633-8c22-e928a12f9286" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": dial tcp 10.133.0.66:8000: connect: connection refused"
Jul 08 17:16:03.677584 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:03.677551    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:16:04.230618 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.230585    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99" podUID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerName="llm-d-routing-sidecar" containerID="cri-o://b360e6657c7fee032a86e9b397b411d7e38fc4027040c4b65d60615a9433fb51" gracePeriod=45
Jul 08 17:16:04.377592 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.377292    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"
Jul 08 17:16:04.483749 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.483723    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:16:04.491618 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.491600    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-kserve-provision-location\") pod \"1ebffd2a-ed17-4633-8c22-e928a12f9286\" (UID: \"1ebffd2a-ed17-4633-8c22-e928a12f9286\") "
Jul 08 17:16:04.491696 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.491655    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-home\") pod \"1ebffd2a-ed17-4633-8c22-e928a12f9286\" (UID: \"1ebffd2a-ed17-4633-8c22-e928a12f9286\") "
Jul 08 17:16:04.491741 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.491695    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-dshm\") pod \"1ebffd2a-ed17-4633-8c22-e928a12f9286\" (UID: \"1ebffd2a-ed17-4633-8c22-e928a12f9286\") "
Jul 08 17:16:04.491741 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.491731    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vgqc\" (UniqueName: \"kubernetes.io/projected/1ebffd2a-ed17-4633-8c22-e928a12f9286-kube-api-access-4vgqc\") pod \"1ebffd2a-ed17-4633-8c22-e928a12f9286\" (UID: \"1ebffd2a-ed17-4633-8c22-e928a12f9286\") "
Jul 08 17:16:04.491851 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.491748    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ebffd2a-ed17-4633-8c22-e928a12f9286-tls-certs\") pod \"1ebffd2a-ed17-4633-8c22-e928a12f9286\" (UID: \"1ebffd2a-ed17-4633-8c22-e928a12f9286\") "
Jul 08 17:16:04.491851 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.491799    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-tmp-dir\") pod \"1ebffd2a-ed17-4633-8c22-e928a12f9286\" (UID: \"1ebffd2a-ed17-4633-8c22-e928a12f9286\") "
Jul 08 17:16:04.491851 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.491825    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-model-cache\") pod \"1ebffd2a-ed17-4633-8c22-e928a12f9286\" (UID: \"1ebffd2a-ed17-4633-8c22-e928a12f9286\") "
Jul 08 17:16:04.492234 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.492208    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-model-cache" (OuterVolumeSpecName: "model-cache") pod "1ebffd2a-ed17-4633-8c22-e928a12f9286" (UID: "1ebffd2a-ed17-4633-8c22-e928a12f9286"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:16:04.492508 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.492479    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-home" (OuterVolumeSpecName: "home") pod "1ebffd2a-ed17-4633-8c22-e928a12f9286" (UID: "1ebffd2a-ed17-4633-8c22-e928a12f9286"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:16:04.493878 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.493849    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ebffd2a-ed17-4633-8c22-e928a12f9286-kube-api-access-4vgqc" (OuterVolumeSpecName: "kube-api-access-4vgqc") pod "1ebffd2a-ed17-4633-8c22-e928a12f9286" (UID: "1ebffd2a-ed17-4633-8c22-e928a12f9286"). InnerVolumeSpecName "kube-api-access-4vgqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:16:04.494221 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.494201    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-dshm" (OuterVolumeSpecName: "dshm") pod "1ebffd2a-ed17-4633-8c22-e928a12f9286" (UID: "1ebffd2a-ed17-4633-8c22-e928a12f9286"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:16:04.494288 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.494224    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ebffd2a-ed17-4633-8c22-e928a12f9286-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1ebffd2a-ed17-4633-8c22-e928a12f9286" (UID: "1ebffd2a-ed17-4633-8c22-e928a12f9286"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:16:04.504329 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.504304    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1ebffd2a-ed17-4633-8c22-e928a12f9286" (UID: "1ebffd2a-ed17-4633-8c22-e928a12f9286"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:16:04.562696 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.562637    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1ebffd2a-ed17-4633-8c22-e928a12f9286" (UID: "1ebffd2a-ed17-4633-8c22-e928a12f9286"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:16:04.593025 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.593000    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-home\") pod \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\" (UID: \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\") "
Jul 08 17:16:04.593143 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.593051    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-dshm\") pod \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\" (UID: \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\") "
Jul 08 17:16:04.593143 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.593075    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-kserve-provision-location\") pod \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\" (UID: \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\") "
Jul 08 17:16:04.593216 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.593175    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-tmp-dir\") pod \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\" (UID: \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\") "
Jul 08 17:16:04.593255 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.593215    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjftb\" (UniqueName: \"kubernetes.io/projected/dce4cf96-d48d-482a-9184-8e2bc1ac3932-kube-api-access-rjftb\") pod \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\" (UID: \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\") "
Jul 08 17:16:04.593298 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.593240    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dce4cf96-d48d-482a-9184-8e2bc1ac3932-tls-certs\") pod \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\" (UID: \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\") "
Jul 08 17:16:04.593353 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.593306    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-model-cache\") pod \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\" (UID: \"dce4cf96-d48d-482a-9184-8e2bc1ac3932\") "
Jul 08 17:16:04.593541 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.593518    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:16:04.593541 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.593542    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vgqc\" (UniqueName: \"kubernetes.io/projected/1ebffd2a-ed17-4633-8c22-e928a12f9286-kube-api-access-4vgqc\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:16:04.593756 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.593556    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ebffd2a-ed17-4633-8c22-e928a12f9286-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:16:04.593756 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.593590    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:16:04.593756 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.593603    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:16:04.593756 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.593615    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-kserve-provision-location\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:16:04.593756 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.593627    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ebffd2a-ed17-4633-8c22-e928a12f9286-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:16:04.593756 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.593725    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-home" (OuterVolumeSpecName: "home") pod "dce4cf96-d48d-482a-9184-8e2bc1ac3932" (UID: "dce4cf96-d48d-482a-9184-8e2bc1ac3932"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:16:04.594073 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.593812    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-model-cache" (OuterVolumeSpecName: "model-cache") pod "dce4cf96-d48d-482a-9184-8e2bc1ac3932" (UID: "dce4cf96-d48d-482a-9184-8e2bc1ac3932"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:16:04.595450 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.595426    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dce4cf96-d48d-482a-9184-8e2bc1ac3932-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dce4cf96-d48d-482a-9184-8e2bc1ac3932" (UID: "dce4cf96-d48d-482a-9184-8e2bc1ac3932"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:16:04.595688 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.595657    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-dshm" (OuterVolumeSpecName: "dshm") pod "dce4cf96-d48d-482a-9184-8e2bc1ac3932" (UID: "dce4cf96-d48d-482a-9184-8e2bc1ac3932"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:16:04.595758 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.595702    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dce4cf96-d48d-482a-9184-8e2bc1ac3932-kube-api-access-rjftb" (OuterVolumeSpecName: "kube-api-access-rjftb") pod "dce4cf96-d48d-482a-9184-8e2bc1ac3932" (UID: "dce4cf96-d48d-482a-9184-8e2bc1ac3932"). InnerVolumeSpecName "kube-api-access-rjftb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:16:04.604245 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.604224    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dce4cf96-d48d-482a-9184-8e2bc1ac3932" (UID: "dce4cf96-d48d-482a-9184-8e2bc1ac3932"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:16:04.659071 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.659038    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dce4cf96-d48d-482a-9184-8e2bc1ac3932" (UID: "dce4cf96-d48d-482a-9184-8e2bc1ac3932"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:16:04.694039 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.694008    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjftb\" (UniqueName: \"kubernetes.io/projected/dce4cf96-d48d-482a-9184-8e2bc1ac3932-kube-api-access-rjftb\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:16:04.694039 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.694031    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dce4cf96-d48d-482a-9184-8e2bc1ac3932-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:16:04.694039 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.694041    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:16:04.694218 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.694050    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:16:04.694218 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.694058    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:16:04.694218 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.694065    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-kserve-provision-location\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:16:04.694218 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.694073    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dce4cf96-d48d-482a-9184-8e2bc1ac3932-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:16:04.955273 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.955241    2565 generic.go:356] "Generic (PLEG): container finished" podID="1ebffd2a-ed17-4633-8c22-e928a12f9286" containerID="03d08a3a1ba64da0ce2b70bdb8bb18d18437dfb38e69599c0f4ad649aa65f603" exitCode=0
Jul 08 17:16:04.955437 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.955285    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"
Jul 08 17:16:04.955437 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.955327    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd" event={"ID":"1ebffd2a-ed17-4633-8c22-e928a12f9286","Type":"ContainerDied","Data":"03d08a3a1ba64da0ce2b70bdb8bb18d18437dfb38e69599c0f4ad649aa65f603"}
Jul 08 17:16:04.955437 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.955376    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd" event={"ID":"1ebffd2a-ed17-4633-8c22-e928a12f9286","Type":"ContainerDied","Data":"69b0522c08c840d0da0ee8700203bf8bb027e216cdc546b8a17c46afd00674ca"}
Jul 08 17:16:04.955437 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.955401    2565 scope.go:117] "RemoveContainer" containerID="03d08a3a1ba64da0ce2b70bdb8bb18d18437dfb38e69599c0f4ad649aa65f603"
Jul 08 17:16:04.957375 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.957350    2565 generic.go:356] "Generic (PLEG): container finished" podID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerID="d0e59f7415b95003e64c32b344f387d82e655a554d48c76060e29239ea79e3c4" exitCode=0
Jul 08 17:16:04.957375 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.957373    2565 generic.go:356] "Generic (PLEG): container finished" podID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerID="b360e6657c7fee032a86e9b397b411d7e38fc4027040c4b65d60615a9433fb51" exitCode=0
Jul 08 17:16:04.957586 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.957381    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99" event={"ID":"dce4cf96-d48d-482a-9184-8e2bc1ac3932","Type":"ContainerDied","Data":"d0e59f7415b95003e64c32b344f387d82e655a554d48c76060e29239ea79e3c4"}
Jul 08 17:16:04.957586 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.957405    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"
Jul 08 17:16:04.957586 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.957434    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99" event={"ID":"dce4cf96-d48d-482a-9184-8e2bc1ac3932","Type":"ContainerDied","Data":"b360e6657c7fee032a86e9b397b411d7e38fc4027040c4b65d60615a9433fb51"}
Jul 08 17:16:04.957586 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.957459    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99" event={"ID":"dce4cf96-d48d-482a-9184-8e2bc1ac3932","Type":"ContainerDied","Data":"6d5510a01c79682512662401884b1e841f36d7613be199035d0c46d76c19be5a"}
Jul 08 17:16:04.964900 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.964883    2565 scope.go:117] "RemoveContainer" containerID="029ce918d05eff69bca384f7e39e0033ada8fab19f86cdf66046f83cd9766ba4"
Jul 08 17:16:04.975623 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.975604    2565 scope.go:117] "RemoveContainer" containerID="03d08a3a1ba64da0ce2b70bdb8bb18d18437dfb38e69599c0f4ad649aa65f603"
Jul 08 17:16:04.975915 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:16:04.975894    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03d08a3a1ba64da0ce2b70bdb8bb18d18437dfb38e69599c0f4ad649aa65f603\": container with ID starting with 03d08a3a1ba64da0ce2b70bdb8bb18d18437dfb38e69599c0f4ad649aa65f603 not found: ID does not exist" containerID="03d08a3a1ba64da0ce2b70bdb8bb18d18437dfb38e69599c0f4ad649aa65f603"
Jul 08 17:16:04.975998 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.975925    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03d08a3a1ba64da0ce2b70bdb8bb18d18437dfb38e69599c0f4ad649aa65f603"} err="failed to get container status \"03d08a3a1ba64da0ce2b70bdb8bb18d18437dfb38e69599c0f4ad649aa65f603\": rpc error: code = NotFound desc = could not find container \"03d08a3a1ba64da0ce2b70bdb8bb18d18437dfb38e69599c0f4ad649aa65f603\": container with ID starting with 03d08a3a1ba64da0ce2b70bdb8bb18d18437dfb38e69599c0f4ad649aa65f603 not found: ID does not exist"
Jul 08 17:16:04.975998 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.975947    2565 scope.go:117] "RemoveContainer" containerID="029ce918d05eff69bca384f7e39e0033ada8fab19f86cdf66046f83cd9766ba4"
Jul 08 17:16:04.976188 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:16:04.976165    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"029ce918d05eff69bca384f7e39e0033ada8fab19f86cdf66046f83cd9766ba4\": container with ID starting with 029ce918d05eff69bca384f7e39e0033ada8fab19f86cdf66046f83cd9766ba4 not found: ID does not exist" containerID="029ce918d05eff69bca384f7e39e0033ada8fab19f86cdf66046f83cd9766ba4"
Jul 08 17:16:04.976236 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.976197    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"029ce918d05eff69bca384f7e39e0033ada8fab19f86cdf66046f83cd9766ba4"} err="failed to get container status \"029ce918d05eff69bca384f7e39e0033ada8fab19f86cdf66046f83cd9766ba4\": rpc error: code = NotFound desc = could not find container \"029ce918d05eff69bca384f7e39e0033ada8fab19f86cdf66046f83cd9766ba4\": container with ID starting with 029ce918d05eff69bca384f7e39e0033ada8fab19f86cdf66046f83cd9766ba4 not found: ID does not exist"
Jul 08 17:16:04.976236 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.976223    2565 scope.go:117] "RemoveContainer" containerID="d0e59f7415b95003e64c32b344f387d82e655a554d48c76060e29239ea79e3c4"
Jul 08 17:16:04.982654 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.982633    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"]
Jul 08 17:16:04.985398 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.985381    2565 scope.go:117] "RemoveContainer" containerID="b486e800825c4e3a801c41a91a13badf8cd8d4f16a68d783d7de7ca1dd7cdf0a"
Jul 08 17:16:04.986833 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.986814    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-prefill-5989gpd"]
Jul 08 17:16:04.993604 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:04.993586    2565 scope.go:117] "RemoveContainer" containerID="b360e6657c7fee032a86e9b397b411d7e38fc4027040c4b65d60615a9433fb51"
Jul 08 17:16:05.001333 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:05.001261    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"]
Jul 08 17:16:05.001400 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:05.001340    2565 scope.go:117] "RemoveContainer" containerID="d0e59f7415b95003e64c32b344f387d82e655a554d48c76060e29239ea79e3c4"
Jul 08 17:16:05.001579 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:16:05.001559    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0e59f7415b95003e64c32b344f387d82e655a554d48c76060e29239ea79e3c4\": container with ID starting with d0e59f7415b95003e64c32b344f387d82e655a554d48c76060e29239ea79e3c4 not found: ID does not exist" containerID="d0e59f7415b95003e64c32b344f387d82e655a554d48c76060e29239ea79e3c4"
Jul 08 17:16:05.001630 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:05.001586    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0e59f7415b95003e64c32b344f387d82e655a554d48c76060e29239ea79e3c4"} err="failed to get container status \"d0e59f7415b95003e64c32b344f387d82e655a554d48c76060e29239ea79e3c4\": rpc error: code = NotFound desc = could not find container \"d0e59f7415b95003e64c32b344f387d82e655a554d48c76060e29239ea79e3c4\": container with ID starting with d0e59f7415b95003e64c32b344f387d82e655a554d48c76060e29239ea79e3c4 not found: ID does not exist"
Jul 08 17:16:05.001630 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:05.001602    2565 scope.go:117] "RemoveContainer" containerID="b486e800825c4e3a801c41a91a13badf8cd8d4f16a68d783d7de7ca1dd7cdf0a"
Jul 08 17:16:05.001826 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:16:05.001811    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b486e800825c4e3a801c41a91a13badf8cd8d4f16a68d783d7de7ca1dd7cdf0a\": container with ID starting with b486e800825c4e3a801c41a91a13badf8cd8d4f16a68d783d7de7ca1dd7cdf0a not found: ID does not exist" containerID="b486e800825c4e3a801c41a91a13badf8cd8d4f16a68d783d7de7ca1dd7cdf0a"
Jul 08 17:16:05.001873 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:05.001831    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b486e800825c4e3a801c41a91a13badf8cd8d4f16a68d783d7de7ca1dd7cdf0a"} err="failed to get container status \"b486e800825c4e3a801c41a91a13badf8cd8d4f16a68d783d7de7ca1dd7cdf0a\": rpc error: code = NotFound desc = could not find container \"b486e800825c4e3a801c41a91a13badf8cd8d4f16a68d783d7de7ca1dd7cdf0a\": container with ID starting with b486e800825c4e3a801c41a91a13badf8cd8d4f16a68d783d7de7ca1dd7cdf0a not found: ID does not exist"
Jul 08 17:16:05.001873 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:05.001844    2565 scope.go:117] "RemoveContainer" containerID="b360e6657c7fee032a86e9b397b411d7e38fc4027040c4b65d60615a9433fb51"
Jul 08 17:16:05.002111 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:16:05.002091    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b360e6657c7fee032a86e9b397b411d7e38fc4027040c4b65d60615a9433fb51\": container with ID starting with b360e6657c7fee032a86e9b397b411d7e38fc4027040c4b65d60615a9433fb51 not found: ID does not exist" containerID="b360e6657c7fee032a86e9b397b411d7e38fc4027040c4b65d60615a9433fb51"
Jul 08 17:16:05.002177 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:05.002120    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b360e6657c7fee032a86e9b397b411d7e38fc4027040c4b65d60615a9433fb51"} err="failed to get container status \"b360e6657c7fee032a86e9b397b411d7e38fc4027040c4b65d60615a9433fb51\": rpc error: code = NotFound desc = could not find container \"b360e6657c7fee032a86e9b397b411d7e38fc4027040c4b65d60615a9433fb51\": container with ID starting with b360e6657c7fee032a86e9b397b411d7e38fc4027040c4b65d60615a9433fb51 not found: ID does not exist"
Jul 08 17:16:05.002177 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:05.002141    2565 scope.go:117] "RemoveContainer" containerID="d0e59f7415b95003e64c32b344f387d82e655a554d48c76060e29239ea79e3c4"
Jul 08 17:16:05.002378 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:05.002350    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0e59f7415b95003e64c32b344f387d82e655a554d48c76060e29239ea79e3c4"} err="failed to get container status \"d0e59f7415b95003e64c32b344f387d82e655a554d48c76060e29239ea79e3c4\": rpc error: code = NotFound desc = could not find container \"d0e59f7415b95003e64c32b344f387d82e655a554d48c76060e29239ea79e3c4\": container with ID starting with d0e59f7415b95003e64c32b344f387d82e655a554d48c76060e29239ea79e3c4 not found: ID does not exist"
Jul 08 17:16:05.002429 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:05.002379    2565 scope.go:117] "RemoveContainer" containerID="b486e800825c4e3a801c41a91a13badf8cd8d4f16a68d783d7de7ca1dd7cdf0a"
Jul 08 17:16:05.002571 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:05.002545    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b486e800825c4e3a801c41a91a13badf8cd8d4f16a68d783d7de7ca1dd7cdf0a"} err="failed to get container status \"b486e800825c4e3a801c41a91a13badf8cd8d4f16a68d783d7de7ca1dd7cdf0a\": rpc error: code = NotFound desc = could not find container \"b486e800825c4e3a801c41a91a13badf8cd8d4f16a68d783d7de7ca1dd7cdf0a\": container with ID starting with b486e800825c4e3a801c41a91a13badf8cd8d4f16a68d783d7de7ca1dd7cdf0a not found: ID does not exist"
Jul 08 17:16:05.002571 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:05.002562    2565 scope.go:117] "RemoveContainer" containerID="b360e6657c7fee032a86e9b397b411d7e38fc4027040c4b65d60615a9433fb51"
Jul 08 17:16:05.002800 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:05.002773    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b360e6657c7fee032a86e9b397b411d7e38fc4027040c4b65d60615a9433fb51"} err="failed to get container status \"b360e6657c7fee032a86e9b397b411d7e38fc4027040c4b65d60615a9433fb51\": rpc error: code = NotFound desc = could not find container \"b360e6657c7fee032a86e9b397b411d7e38fc4027040c4b65d60615a9433fb51\": container with ID starting with b360e6657c7fee032a86e9b397b411d7e38fc4027040c4b65d60615a9433fb51 not found: ID does not exist"
Jul 08 17:16:05.011757 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:05.011738    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-50bc673d-kserve-59596f6d4fsgc99"]
Jul 08 17:16:05.822290 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:05.822256    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ebffd2a-ed17-4633-8c22-e928a12f9286" path="/var/lib/kubelet/pods/1ebffd2a-ed17-4633-8c22-e928a12f9286/volumes"
Jul 08 17:16:05.822767 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:05.822740    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" path="/var/lib/kubelet/pods/dce4cf96-d48d-482a-9184-8e2bc1ac3932/volumes"
Jul 08 17:16:18.971463 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:18.971417    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6" podUID="32cee549-134c-4fad-8b17-ed32953bde74" containerName="main" containerID="cri-o://cbb7c8445c0c9b7f5538083d5fa3a151037b45e1f2715e3d6054a24c174bb040" gracePeriod=45
Jul 08 17:16:19.070770 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.070735    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"]
Jul 08 17:16:19.071194 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.071146    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerName="llm-d-routing-sidecar"
Jul 08 17:16:19.071194 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.071165    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerName="llm-d-routing-sidecar"
Jul 08 17:16:19.071194 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.071187    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerName="storage-initializer"
Jul 08 17:16:19.071194 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.071196    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerName="storage-initializer"
Jul 08 17:16:19.071503 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.071214    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ebffd2a-ed17-4633-8c22-e928a12f9286" containerName="storage-initializer"
Jul 08 17:16:19.071503 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.071226    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ebffd2a-ed17-4633-8c22-e928a12f9286" containerName="storage-initializer"
Jul 08 17:16:19.071503 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.071238    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ebffd2a-ed17-4633-8c22-e928a12f9286" containerName="main"
Jul 08 17:16:19.071503 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.071247    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ebffd2a-ed17-4633-8c22-e928a12f9286" containerName="main"
Jul 08 17:16:19.071503 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.071266    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerName="main"
Jul 08 17:16:19.071503 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.071274    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerName="main"
Jul 08 17:16:19.071503 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.071377    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerName="main"
Jul 08 17:16:19.071503 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.071391    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="dce4cf96-d48d-482a-9184-8e2bc1ac3932" containerName="llm-d-routing-sidecar"
Jul 08 17:16:19.071503 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.071405    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ebffd2a-ed17-4633-8c22-e928a12f9286" containerName="main"
Jul 08 17:16:19.075411 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.075390    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:16:19.080134 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.080114    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-self-signed-certs\""
Jul 08 17:16:19.080267 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.080251    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"custom-route-timeout-pd-test-kserve-dockercfg-j5h7l\""
Jul 08 17:16:19.085991 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.085971    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"]
Jul 08 17:16:19.088476 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.088337    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"
Jul 08 17:16:19.097384 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.097362    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"]
Jul 08 17:16:19.106081 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.106058    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"]
Jul 08 17:16:19.122123 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.122093    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:16:19.216392 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.216359    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456\" (UID: \"7ef5d329-0988-41ba-847d-34c0cd92d6be\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"
Jul 08 17:16:19.216392 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.216402    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9c8d\" (UniqueName: \"kubernetes.io/projected/7ef5d329-0988-41ba-847d-34c0cd92d6be-kube-api-access-v9c8d\") pod \"custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456\" (UID: \"7ef5d329-0988-41ba-847d-34c0cd92d6be\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"
Jul 08 17:16:19.216639 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.216424    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456\" (UID: \"7ef5d329-0988-41ba-847d-34c0cd92d6be\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"
Jul 08 17:16:19.216639 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.216446    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-67588db874-f6b8t\" (UID: \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:16:19.216639 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.216491    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-dshm\") pod \"custom-route-timeout-pd-test-kserve-67588db874-f6b8t\" (UID: \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:16:19.216639 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.216514    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ef5d329-0988-41ba-847d-34c0cd92d6be-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456\" (UID: \"7ef5d329-0988-41ba-847d-34c0cd92d6be\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"
Jul 08 17:16:19.216639 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.216573    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-model-cache\") pod \"custom-route-timeout-pd-test-kserve-67588db874-f6b8t\" (UID: \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:16:19.216639 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.216594    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e1ec99-cae2-42e4-908b-c7bc01986b1e-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-67588db874-f6b8t\" (UID: \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:16:19.216639 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.216622    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456\" (UID: \"7ef5d329-0988-41ba-847d-34c0cd92d6be\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"
Jul 08 17:16:19.216934 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.216651    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbjfp\" (UniqueName: \"kubernetes.io/projected/51e1ec99-cae2-42e4-908b-c7bc01986b1e-kube-api-access-zbjfp\") pod \"custom-route-timeout-pd-test-kserve-67588db874-f6b8t\" (UID: \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:16:19.216934 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.216669    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456\" (UID: \"7ef5d329-0988-41ba-847d-34c0cd92d6be\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"
Jul 08 17:16:19.216934 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.216744    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-home\") pod \"custom-route-timeout-pd-test-kserve-67588db874-f6b8t\" (UID: \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:16:19.216934 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.216809    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-67588db874-f6b8t\" (UID: \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:16:19.216934 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.216854    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456\" (UID: \"7ef5d329-0988-41ba-847d-34c0cd92d6be\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"
Jul 08 17:16:19.318232 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.318146    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbjfp\" (UniqueName: \"kubernetes.io/projected/51e1ec99-cae2-42e4-908b-c7bc01986b1e-kube-api-access-zbjfp\") pod \"custom-route-timeout-pd-test-kserve-67588db874-f6b8t\" (UID: \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:16:19.318232 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.318181    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456\" (UID: \"7ef5d329-0988-41ba-847d-34c0cd92d6be\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"
Jul 08 17:16:19.318232 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.318202    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-home\") pod \"custom-route-timeout-pd-test-kserve-67588db874-f6b8t\" (UID: \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:16:19.318662 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.318488    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-67588db874-f6b8t\" (UID: \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:16:19.318662 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.318553    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456\" (UID: \"7ef5d329-0988-41ba-847d-34c0cd92d6be\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"
Jul 08 17:16:19.318662 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.318587    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-home\") pod \"custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456\" (UID: \"7ef5d329-0988-41ba-847d-34c0cd92d6be\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"
Jul 08 17:16:19.318662 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.318617    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456\" (UID: \"7ef5d329-0988-41ba-847d-34c0cd92d6be\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"
Jul 08 17:16:19.318662 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.318645    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9c8d\" (UniqueName: \"kubernetes.io/projected/7ef5d329-0988-41ba-847d-34c0cd92d6be-kube-api-access-v9c8d\") pod \"custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456\" (UID: \"7ef5d329-0988-41ba-847d-34c0cd92d6be\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"
Jul 08 17:16:19.318960 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.318687    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456\" (UID: \"7ef5d329-0988-41ba-847d-34c0cd92d6be\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"
Jul 08 17:16:19.318960 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.318716    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-67588db874-f6b8t\" (UID: \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:16:19.318960 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.318751    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-dshm\") pod \"custom-route-timeout-pd-test-kserve-67588db874-f6b8t\" (UID: \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:16:19.318960 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.318784    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ef5d329-0988-41ba-847d-34c0cd92d6be-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456\" (UID: \"7ef5d329-0988-41ba-847d-34c0cd92d6be\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"
Jul 08 17:16:19.318960 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.318803    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-67588db874-f6b8t\" (UID: \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:16:19.318960 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.318833    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-model-cache\") pod \"custom-route-timeout-pd-test-kserve-67588db874-f6b8t\" (UID: \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:16:19.318960 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.318865    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e1ec99-cae2-42e4-908b-c7bc01986b1e-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-67588db874-f6b8t\" (UID: \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:16:19.318960 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.318893    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-model-cache\") pod \"custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456\" (UID: \"7ef5d329-0988-41ba-847d-34c0cd92d6be\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"
Jul 08 17:16:19.318960 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.318912    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456\" (UID: \"7ef5d329-0988-41ba-847d-34c0cd92d6be\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"
Jul 08 17:16:19.319456 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.318910    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-kserve-provision-location\") pod \"custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456\" (UID: \"7ef5d329-0988-41ba-847d-34c0cd92d6be\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"
Jul 08 17:16:19.319456 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.318982    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456\" (UID: \"7ef5d329-0988-41ba-847d-34c0cd92d6be\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"
Jul 08 17:16:19.319456 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.318750    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-home\") pod \"custom-route-timeout-pd-test-kserve-67588db874-f6b8t\" (UID: \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:16:19.319456 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.319136    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-tmp-dir\") pod \"custom-route-timeout-pd-test-kserve-67588db874-f6b8t\" (UID: \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:16:19.319456 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.319235    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-model-cache\") pod \"custom-route-timeout-pd-test-kserve-67588db874-f6b8t\" (UID: \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:16:19.321457 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.321431    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-dshm\") pod \"custom-route-timeout-pd-test-kserve-67588db874-f6b8t\" (UID: \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:16:19.321583 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.321486    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ef5d329-0988-41ba-847d-34c0cd92d6be-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456\" (UID: \"7ef5d329-0988-41ba-847d-34c0cd92d6be\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"
Jul 08 17:16:19.321583 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.321494    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-dshm\") pod \"custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456\" (UID: \"7ef5d329-0988-41ba-847d-34c0cd92d6be\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"
Jul 08 17:16:19.321762 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.321745    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e1ec99-cae2-42e4-908b-c7bc01986b1e-tls-certs\") pod \"custom-route-timeout-pd-test-kserve-67588db874-f6b8t\" (UID: \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:16:19.327968 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.327949    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbjfp\" (UniqueName: \"kubernetes.io/projected/51e1ec99-cae2-42e4-908b-c7bc01986b1e-kube-api-access-zbjfp\") pod \"custom-route-timeout-pd-test-kserve-67588db874-f6b8t\" (UID: \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:16:19.330322 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.330301    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9c8d\" (UniqueName: \"kubernetes.io/projected/7ef5d329-0988-41ba-847d-34c0cd92d6be-kube-api-access-v9c8d\") pod \"custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456\" (UID: \"7ef5d329-0988-41ba-847d-34c0cd92d6be\") " pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"
Jul 08 17:16:19.386326 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.386282    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:16:19.400016 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.399977    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"
Jul 08 17:16:19.472127 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.471226    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:16:19.568185 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.568142    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"]
Jul 08 17:16:19.576976 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:16:19.576949    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e1ec99_cae2_42e4_908b_c7bc01986b1e.slice/crio-a23383da6c3d9d565fabe17f52debea223ab3499eceffebec33e0b82f75e4af7 WatchSource:0}: Error finding container a23383da6c3d9d565fabe17f52debea223ab3499eceffebec33e0b82f75e4af7: Status 404 returned error can't find the container with id a23383da6c3d9d565fabe17f52debea223ab3499eceffebec33e0b82f75e4af7
Jul 08 17:16:19.587383 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.587338    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"]
Jul 08 17:16:19.589963 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:16:19.589939    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef5d329_0988_41ba_847d_34c0cd92d6be.slice/crio-5b976265fb1b046bd050c98d730c920e65713266e6038a46e3f6011a74ecc78b WatchSource:0}: Error finding container 5b976265fb1b046bd050c98d730c920e65713266e6038a46e3f6011a74ecc78b: Status 404 returned error can't find the container with id 5b976265fb1b046bd050c98d730c920e65713266e6038a46e3f6011a74ecc78b
Jul 08 17:16:19.745902 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.745876    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"
Jul 08 17:16:19.824624 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.824540    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-dshm\") pod \"32cee549-134c-4fad-8b17-ed32953bde74\" (UID: \"32cee549-134c-4fad-8b17-ed32953bde74\") "
Jul 08 17:16:19.824803 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.824633    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-tmp-dir\") pod \"32cee549-134c-4fad-8b17-ed32953bde74\" (UID: \"32cee549-134c-4fad-8b17-ed32953bde74\") "
Jul 08 17:16:19.824803 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.824663    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t9tq9\" (UniqueName: \"kubernetes.io/projected/32cee549-134c-4fad-8b17-ed32953bde74-kube-api-access-t9tq9\") pod \"32cee549-134c-4fad-8b17-ed32953bde74\" (UID: \"32cee549-134c-4fad-8b17-ed32953bde74\") "
Jul 08 17:16:19.824803 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.824714    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-kserve-provision-location\") pod \"32cee549-134c-4fad-8b17-ed32953bde74\" (UID: \"32cee549-134c-4fad-8b17-ed32953bde74\") "
Jul 08 17:16:19.824803 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.824747    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-home\") pod \"32cee549-134c-4fad-8b17-ed32953bde74\" (UID: \"32cee549-134c-4fad-8b17-ed32953bde74\") "
Jul 08 17:16:19.824803 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.824784    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32cee549-134c-4fad-8b17-ed32953bde74-tls-certs\") pod \"32cee549-134c-4fad-8b17-ed32953bde74\" (UID: \"32cee549-134c-4fad-8b17-ed32953bde74\") "
Jul 08 17:16:19.825072 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.824826    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-model-cache\") pod \"32cee549-134c-4fad-8b17-ed32953bde74\" (UID: \"32cee549-134c-4fad-8b17-ed32953bde74\") "
Jul 08 17:16:19.825479 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.825234    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-model-cache" (OuterVolumeSpecName: "model-cache") pod "32cee549-134c-4fad-8b17-ed32953bde74" (UID: "32cee549-134c-4fad-8b17-ed32953bde74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:16:19.825599 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.825546    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-home" (OuterVolumeSpecName: "home") pod "32cee549-134c-4fad-8b17-ed32953bde74" (UID: "32cee549-134c-4fad-8b17-ed32953bde74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:16:19.827176 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.827151    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-dshm" (OuterVolumeSpecName: "dshm") pod "32cee549-134c-4fad-8b17-ed32953bde74" (UID: "32cee549-134c-4fad-8b17-ed32953bde74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:16:19.827733 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.827711    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32cee549-134c-4fad-8b17-ed32953bde74-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32cee549-134c-4fad-8b17-ed32953bde74" (UID: "32cee549-134c-4fad-8b17-ed32953bde74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:16:19.828951 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.828931    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32cee549-134c-4fad-8b17-ed32953bde74-kube-api-access-t9tq9" (OuterVolumeSpecName: "kube-api-access-t9tq9") pod "32cee549-134c-4fad-8b17-ed32953bde74" (UID: "32cee549-134c-4fad-8b17-ed32953bde74"). InnerVolumeSpecName "kube-api-access-t9tq9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:16:19.837485 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.837462    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "32cee549-134c-4fad-8b17-ed32953bde74" (UID: "32cee549-134c-4fad-8b17-ed32953bde74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:16:19.898462 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.898415    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "32cee549-134c-4fad-8b17-ed32953bde74" (UID: "32cee549-134c-4fad-8b17-ed32953bde74"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:16:19.926197 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.926169    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-kserve-provision-location\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:16:19.926197 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.926194    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:16:19.926197 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.926205    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32cee549-134c-4fad-8b17-ed32953bde74-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:16:19.926437 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.926214    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:16:19.926437 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.926226    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:16:19.926437 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.926237    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cee549-134c-4fad-8b17-ed32953bde74-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:16:19.926437 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:19.926249    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9tq9\" (UniqueName: \"kubernetes.io/projected/32cee549-134c-4fad-8b17-ed32953bde74-kube-api-access-t9tq9\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:16:20.042429 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:20.042389    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" event={"ID":"51e1ec99-cae2-42e4-908b-c7bc01986b1e","Type":"ContainerStarted","Data":"d3cb180f19e4a58e872ad62f0c68ab9b7a1c5f50b88368b862e125ef161e3d8c"}
Jul 08 17:16:20.042899 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:20.042464    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:16:20.042899 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:20.042505    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" event={"ID":"51e1ec99-cae2-42e4-908b-c7bc01986b1e","Type":"ContainerStarted","Data":"a23383da6c3d9d565fabe17f52debea223ab3499eceffebec33e0b82f75e4af7"}
Jul 08 17:16:20.044260 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:20.044237    2565 generic.go:356] "Generic (PLEG): container finished" podID="32cee549-134c-4fad-8b17-ed32953bde74" containerID="cbb7c8445c0c9b7f5538083d5fa3a151037b45e1f2715e3d6054a24c174bb040" exitCode=0
Jul 08 17:16:20.044381 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:20.044285    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"
Jul 08 17:16:20.044381 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:20.044315    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6" event={"ID":"32cee549-134c-4fad-8b17-ed32953bde74","Type":"ContainerDied","Data":"cbb7c8445c0c9b7f5538083d5fa3a151037b45e1f2715e3d6054a24c174bb040"}
Jul 08 17:16:20.044381 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:20.044373    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6" event={"ID":"32cee549-134c-4fad-8b17-ed32953bde74","Type":"ContainerDied","Data":"6332786b3d1ae2debcecc282e507283042b64782384c4a2e1f9d57ef68beb193"}
Jul 08 17:16:20.044535 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:20.044408    2565 scope.go:117] "RemoveContainer" containerID="cbb7c8445c0c9b7f5538083d5fa3a151037b45e1f2715e3d6054a24c174bb040"
Jul 08 17:16:20.046019 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:20.045989    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456" event={"ID":"7ef5d329-0988-41ba-847d-34c0cd92d6be","Type":"ContainerStarted","Data":"82975f5e225cca6c6e6dd7178887fa53abb523b4ac69425f051e5b90f6264a86"}
Jul 08 17:16:20.046136 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:20.046030    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456" event={"ID":"7ef5d329-0988-41ba-847d-34c0cd92d6be","Type":"ContainerStarted","Data":"5b976265fb1b046bd050c98d730c920e65713266e6038a46e3f6011a74ecc78b"}
Jul 08 17:16:20.059326 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:20.059303    2565 scope.go:117] "RemoveContainer" containerID="dadad4379ae59be752be7ea4886e14502af9433c3493aa27851dae3c3144aa2b"
Jul 08 17:16:20.070035 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:20.070010    2565 scope.go:117] "RemoveContainer" containerID="cbb7c8445c0c9b7f5538083d5fa3a151037b45e1f2715e3d6054a24c174bb040"
Jul 08 17:16:20.070338 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:16:20.070312    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbb7c8445c0c9b7f5538083d5fa3a151037b45e1f2715e3d6054a24c174bb040\": container with ID starting with cbb7c8445c0c9b7f5538083d5fa3a151037b45e1f2715e3d6054a24c174bb040 not found: ID does not exist" containerID="cbb7c8445c0c9b7f5538083d5fa3a151037b45e1f2715e3d6054a24c174bb040"
Jul 08 17:16:20.070442 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:20.070350    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbb7c8445c0c9b7f5538083d5fa3a151037b45e1f2715e3d6054a24c174bb040"} err="failed to get container status \"cbb7c8445c0c9b7f5538083d5fa3a151037b45e1f2715e3d6054a24c174bb040\": rpc error: code = NotFound desc = could not find container \"cbb7c8445c0c9b7f5538083d5fa3a151037b45e1f2715e3d6054a24c174bb040\": container with ID starting with cbb7c8445c0c9b7f5538083d5fa3a151037b45e1f2715e3d6054a24c174bb040 not found: ID does not exist"
Jul 08 17:16:20.070442 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:20.070386    2565 scope.go:117] "RemoveContainer" containerID="dadad4379ae59be752be7ea4886e14502af9433c3493aa27851dae3c3144aa2b"
Jul 08 17:16:20.070705 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:16:20.070650    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dadad4379ae59be752be7ea4886e14502af9433c3493aa27851dae3c3144aa2b\": container with ID starting with dadad4379ae59be752be7ea4886e14502af9433c3493aa27851dae3c3144aa2b not found: ID does not exist" containerID="dadad4379ae59be752be7ea4886e14502af9433c3493aa27851dae3c3144aa2b"
Jul 08 17:16:20.070803 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:20.070701    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dadad4379ae59be752be7ea4886e14502af9433c3493aa27851dae3c3144aa2b"} err="failed to get container status \"dadad4379ae59be752be7ea4886e14502af9433c3493aa27851dae3c3144aa2b\": rpc error: code = NotFound desc = could not find container \"dadad4379ae59be752be7ea4886e14502af9433c3493aa27851dae3c3144aa2b\": container with ID starting with dadad4379ae59be752be7ea4886e14502af9433c3493aa27851dae3c3144aa2b not found: ID does not exist"
Jul 08 17:16:20.093891 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:20.093866    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"]
Jul 08 17:16:20.098458 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:20.098435    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-knxs6"]
Jul 08 17:16:21.057150 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:21.057115    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" event={"ID":"51e1ec99-cae2-42e4-908b-c7bc01986b1e","Type":"ContainerStarted","Data":"a74a774bc30c381a4c1a970ec4ffa111652c602d21116b4d641e4ad02b38ffab"}
Jul 08 17:16:21.823822 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:21.823786    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32cee549-134c-4fad-8b17-ed32953bde74" path="/var/lib/kubelet/pods/32cee549-134c-4fad-8b17-ed32953bde74/volumes"
Jul 08 17:16:25.089810 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:25.089775    2565 generic.go:356] "Generic (PLEG): container finished" podID="7ef5d329-0988-41ba-847d-34c0cd92d6be" containerID="82975f5e225cca6c6e6dd7178887fa53abb523b4ac69425f051e5b90f6264a86" exitCode=0
Jul 08 17:16:25.090176 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:25.089839    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456" event={"ID":"7ef5d329-0988-41ba-847d-34c0cd92d6be","Type":"ContainerDied","Data":"82975f5e225cca6c6e6dd7178887fa53abb523b4ac69425f051e5b90f6264a86"}
Jul 08 17:16:26.099050 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:26.099018    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456" event={"ID":"7ef5d329-0988-41ba-847d-34c0cd92d6be","Type":"ContainerStarted","Data":"77576122de76468cd563ff6dc9427b5c706fe92b8927ab9db132b560aae79b34"}
Jul 08 17:16:26.123038 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:26.122972    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456" podStartSLOduration=2.257819103 podStartE2EDuration="7.12295037s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.865131267s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:16:26.119063604 +0000 UTC m=+2522.832425030" watchObservedRunningTime="2026-07-08 17:16:26.12295037 +0000 UTC m=+2522.836311798"
Jul 08 17:16:27.108158 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:27.108125    2565 generic.go:356] "Generic (PLEG): container finished" podID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerID="a74a774bc30c381a4c1a970ec4ffa111652c602d21116b4d641e4ad02b38ffab" exitCode=0
Jul 08 17:16:27.108549 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:27.108201    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" event={"ID":"51e1ec99-cae2-42e4-908b-c7bc01986b1e","Type":"ContainerDied","Data":"a74a774bc30c381a4c1a970ec4ffa111652c602d21116b4d641e4ad02b38ffab"}
Jul 08 17:16:28.116895 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:28.116857    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" event={"ID":"51e1ec99-cae2-42e4-908b-c7bc01986b1e","Type":"ContainerStarted","Data":"254ae93d4101a2d7cd85b3c57770bb28c13f7fa228d663209628f508012ebe8b"}
Jul 08 17:16:28.147955 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:28.147881    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" podStartSLOduration=2.808684241 podStartE2EDuration="9.147860328s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.339176087s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:16:28.146378972 +0000 UTC m=+2524.859740428" watchObservedRunningTime="2026-07-08 17:16:28.147860328 +0000 UTC m=+2524.861221757"
Jul 08 17:16:29.387403 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:29.387361    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:16:29.387835 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:29.387459    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:16:29.388874 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:29.388829    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8001/health\": dial tcp 10.133.0.68:8001: connect: connection refused"
Jul 08 17:16:29.400183 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:29.400157    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"
Jul 08 17:16:29.400326 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:29.400219    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"
Jul 08 17:16:29.401396 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:29.401365    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456" podUID="7ef5d329-0988-41ba-847d-34c0cd92d6be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Jul 08 17:16:29.402086 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:29.402068    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:16:30.290475 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.290437    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"]
Jul 08 17:16:30.290896 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.290882    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32cee549-134c-4fad-8b17-ed32953bde74" containerName="storage-initializer"
Jul 08 17:16:30.290948 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.290898    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="32cee549-134c-4fad-8b17-ed32953bde74" containerName="storage-initializer"
Jul 08 17:16:30.290948 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.290915    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32cee549-134c-4fad-8b17-ed32953bde74" containerName="main"
Jul 08 17:16:30.290948 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.290921    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="32cee549-134c-4fad-8b17-ed32953bde74" containerName="main"
Jul 08 17:16:30.291043 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.290988    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="32cee549-134c-4fad-8b17-ed32953bde74" containerName="main"
Jul 08 17:16:30.294715 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.294689    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"
Jul 08 17:16:30.297582 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.297553    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"stop-feature-test-kserve-self-signed-certs\""
Jul 08 17:16:30.307908 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.307875    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"]
Jul 08 17:16:30.335054 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.335011    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-home\") pod \"stop-feature-test-kserve-7c7845f79f-tfvwg\" (UID: \"328715cc-4fc0-4fa6-9803-bfe419543da5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"
Jul 08 17:16:30.335054 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.335057    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/328715cc-4fc0-4fa6-9803-bfe419543da5-tls-certs\") pod \"stop-feature-test-kserve-7c7845f79f-tfvwg\" (UID: \"328715cc-4fc0-4fa6-9803-bfe419543da5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"
Jul 08 17:16:30.335277 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.335112    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-dshm\") pod \"stop-feature-test-kserve-7c7845f79f-tfvwg\" (UID: \"328715cc-4fc0-4fa6-9803-bfe419543da5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"
Jul 08 17:16:30.335277 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.335164    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-tmp-dir\") pod \"stop-feature-test-kserve-7c7845f79f-tfvwg\" (UID: \"328715cc-4fc0-4fa6-9803-bfe419543da5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"
Jul 08 17:16:30.335277 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.335202    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-model-cache\") pod \"stop-feature-test-kserve-7c7845f79f-tfvwg\" (UID: \"328715cc-4fc0-4fa6-9803-bfe419543da5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"
Jul 08 17:16:30.335382 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.335303    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r9bs\" (UniqueName: \"kubernetes.io/projected/328715cc-4fc0-4fa6-9803-bfe419543da5-kube-api-access-4r9bs\") pod \"stop-feature-test-kserve-7c7845f79f-tfvwg\" (UID: \"328715cc-4fc0-4fa6-9803-bfe419543da5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"
Jul 08 17:16:30.335382 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.335368    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-kserve-provision-location\") pod \"stop-feature-test-kserve-7c7845f79f-tfvwg\" (UID: \"328715cc-4fc0-4fa6-9803-bfe419543da5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"
Jul 08 17:16:30.364955 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.364914    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:16:30.436391 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.436351    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-tmp-dir\") pod \"stop-feature-test-kserve-7c7845f79f-tfvwg\" (UID: \"328715cc-4fc0-4fa6-9803-bfe419543da5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"
Jul 08 17:16:30.436858 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.436429    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-model-cache\") pod \"stop-feature-test-kserve-7c7845f79f-tfvwg\" (UID: \"328715cc-4fc0-4fa6-9803-bfe419543da5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"
Jul 08 17:16:30.436858 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.436499    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r9bs\" (UniqueName: \"kubernetes.io/projected/328715cc-4fc0-4fa6-9803-bfe419543da5-kube-api-access-4r9bs\") pod \"stop-feature-test-kserve-7c7845f79f-tfvwg\" (UID: \"328715cc-4fc0-4fa6-9803-bfe419543da5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"
Jul 08 17:16:30.436858 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.436548    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-kserve-provision-location\") pod \"stop-feature-test-kserve-7c7845f79f-tfvwg\" (UID: \"328715cc-4fc0-4fa6-9803-bfe419543da5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"
Jul 08 17:16:30.436858 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.436593    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-home\") pod \"stop-feature-test-kserve-7c7845f79f-tfvwg\" (UID: \"328715cc-4fc0-4fa6-9803-bfe419543da5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"
Jul 08 17:16:30.436858 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.436624    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/328715cc-4fc0-4fa6-9803-bfe419543da5-tls-certs\") pod \"stop-feature-test-kserve-7c7845f79f-tfvwg\" (UID: \"328715cc-4fc0-4fa6-9803-bfe419543da5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"
Jul 08 17:16:30.436858 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.436658    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-dshm\") pod \"stop-feature-test-kserve-7c7845f79f-tfvwg\" (UID: \"328715cc-4fc0-4fa6-9803-bfe419543da5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"
Jul 08 17:16:30.437210 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.437177    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-tmp-dir\") pod \"stop-feature-test-kserve-7c7845f79f-tfvwg\" (UID: \"328715cc-4fc0-4fa6-9803-bfe419543da5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"
Jul 08 17:16:30.437210 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.437201    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-kserve-provision-location\") pod \"stop-feature-test-kserve-7c7845f79f-tfvwg\" (UID: \"328715cc-4fc0-4fa6-9803-bfe419543da5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"
Jul 08 17:16:30.437408 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.437255    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-model-cache\") pod \"stop-feature-test-kserve-7c7845f79f-tfvwg\" (UID: \"328715cc-4fc0-4fa6-9803-bfe419543da5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"
Jul 08 17:16:30.437408 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.437309    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-home\") pod \"stop-feature-test-kserve-7c7845f79f-tfvwg\" (UID: \"328715cc-4fc0-4fa6-9803-bfe419543da5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"
Jul 08 17:16:30.439060 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.439037    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-dshm\") pod \"stop-feature-test-kserve-7c7845f79f-tfvwg\" (UID: \"328715cc-4fc0-4fa6-9803-bfe419543da5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"
Jul 08 17:16:30.439710 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.439670    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/328715cc-4fc0-4fa6-9803-bfe419543da5-tls-certs\") pod \"stop-feature-test-kserve-7c7845f79f-tfvwg\" (UID: \"328715cc-4fc0-4fa6-9803-bfe419543da5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"
Jul 08 17:16:30.448989 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.448961    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r9bs\" (UniqueName: \"kubernetes.io/projected/328715cc-4fc0-4fa6-9803-bfe419543da5-kube-api-access-4r9bs\") pod \"stop-feature-test-kserve-7c7845f79f-tfvwg\" (UID: \"328715cc-4fc0-4fa6-9803-bfe419543da5\") " pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"
Jul 08 17:16:30.607745 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.607624    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"
Jul 08 17:16:30.690844 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.690727    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:16:30.777925 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:30.777892    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"]
Jul 08 17:16:30.778590 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:16:30.778556    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod328715cc_4fc0_4fa6_9803_bfe419543da5.slice/crio-014d135232e6eef94296f9697a9741e0e85a21cf1f51c26f18cd2d7281f83330 WatchSource:0}: Error finding container 014d135232e6eef94296f9697a9741e0e85a21cf1f51c26f18cd2d7281f83330: Status 404 returned error can't find the container with id 014d135232e6eef94296f9697a9741e0e85a21cf1f51c26f18cd2d7281f83330
Jul 08 17:16:31.149286 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:31.149196    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg" event={"ID":"328715cc-4fc0-4fa6-9803-bfe419543da5","Type":"ContainerStarted","Data":"70ce2fc7fd56ffbd591150fcf9934ce922fdcfe8652aa9c97122987290e2f00f"}
Jul 08 17:16:31.149286 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:31.149262    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg" event={"ID":"328715cc-4fc0-4fa6-9803-bfe419543da5","Type":"ContainerStarted","Data":"014d135232e6eef94296f9697a9741e0e85a21cf1f51c26f18cd2d7281f83330"}
Jul 08 17:16:36.198490 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:36.198449    2565 generic.go:356] "Generic (PLEG): container finished" podID="328715cc-4fc0-4fa6-9803-bfe419543da5" containerID="70ce2fc7fd56ffbd591150fcf9934ce922fdcfe8652aa9c97122987290e2f00f" exitCode=0
Jul 08 17:16:36.198955 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:36.198524    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg" event={"ID":"328715cc-4fc0-4fa6-9803-bfe419543da5","Type":"ContainerDied","Data":"70ce2fc7fd56ffbd591150fcf9934ce922fdcfe8652aa9c97122987290e2f00f"}
Jul 08 17:16:37.206363 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:37.206327    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg" event={"ID":"328715cc-4fc0-4fa6-9803-bfe419543da5","Type":"ContainerStarted","Data":"d376691e08b9024b6118a8fffdba1a11e8b9e2f5bc0ade8b475f3185c7b655c9"}
Jul 08 17:16:37.242430 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:37.242343    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg" podStartSLOduration=2.098699605 podStartE2EDuration="7.242315291s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.143615686s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:16:37.23752002 +0000 UTC m=+2533.950881472" watchObservedRunningTime="2026-07-08 17:16:37.242315291 +0000 UTC m=+2533.955676719"
Jul 08 17:16:37.607845 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:37.607745    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"
Jul 08 17:16:39.387327 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:39.387274    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8001/health\": dial tcp 10.133.0.68:8001: connect: connection refused"
Jul 08 17:16:39.401071 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:39.401029    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456" podUID="7ef5d329-0988-41ba-847d-34c0cd92d6be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Jul 08 17:16:40.608733 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:40.608667    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"
Jul 08 17:16:40.610354 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:40.610317    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg" podUID="328715cc-4fc0-4fa6-9803-bfe419543da5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 08 17:16:43.381600 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:43.381554    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:16:49.387812 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:49.387769    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8001/health\": dial tcp 10.133.0.68:8001: connect: connection refused"
Jul 08 17:16:49.400398 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:49.400358    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456" podUID="7ef5d329-0988-41ba-847d-34c0cd92d6be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Jul 08 17:16:49.670088 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:49.669998    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:16:50.608884 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:50.608824    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg" podUID="328715cc-4fc0-4fa6-9803-bfe419543da5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 08 17:16:59.386983 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:59.386925    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8001/health\": dial tcp 10.133.0.68:8001: connect: connection refused"
Jul 08 17:16:59.400953 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:16:59.400907    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456" podUID="7ef5d329-0988-41ba-847d-34c0cd92d6be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Jul 08 17:17:00.608551 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:17:00.608505    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg" podUID="328715cc-4fc0-4fa6-9803-bfe419543da5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 08 17:17:09.387242 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:17:09.387183    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8001/health\": dial tcp 10.133.0.68:8001: connect: connection refused"
Jul 08 17:17:09.401050 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:17:09.401000    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456" podUID="7ef5d329-0988-41ba-847d-34c0cd92d6be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Jul 08 17:17:10.608530 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:17:10.608479    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg" podUID="328715cc-4fc0-4fa6-9803-bfe419543da5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 08 17:17:19.387570 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:17:19.387514    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8001/health\": dial tcp 10.133.0.68:8001: connect: connection refused"
Jul 08 17:17:19.400596 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:17:19.400553    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456" podUID="7ef5d329-0988-41ba-847d-34c0cd92d6be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Jul 08 17:17:20.608618 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:17:20.608575    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg" podUID="328715cc-4fc0-4fa6-9803-bfe419543da5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 08 17:17:29.387232 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:17:29.387179    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8001/health\": dial tcp 10.133.0.68:8001: connect: connection refused"
Jul 08 17:17:29.400828 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:17:29.400784    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456" podUID="7ef5d329-0988-41ba-847d-34c0cd92d6be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Jul 08 17:17:30.608940 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:17:30.608892    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg" podUID="328715cc-4fc0-4fa6-9803-bfe419543da5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 08 17:17:36.702828 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:17:36.702777    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:17:39.387014 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:17:39.386963    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8001/health\": dial tcp 10.133.0.68:8001: connect: connection refused"
Jul 08 17:17:39.400791 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:17:39.400739    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456" podUID="7ef5d329-0988-41ba-847d-34c0cd92d6be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Jul 08 17:17:40.608353 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:17:40.608306    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg" podUID="328715cc-4fc0-4fa6-9803-bfe419543da5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 08 17:17:49.387386 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:17:49.387270    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8001/health\": dial tcp 10.133.0.68:8001: connect: connection refused"
Jul 08 17:17:49.400594 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:17:49.400553    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456" podUID="7ef5d329-0988-41ba-847d-34c0cd92d6be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Jul 08 17:17:50.609172 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:17:50.609117    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg" podUID="328715cc-4fc0-4fa6-9803-bfe419543da5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 08 17:17:57.118098 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:17:57.118057    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:17:59.387069 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:17:59.387011    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8001/health\": dial tcp 10.133.0.68:8001: connect: connection refused"
Jul 08 17:17:59.400610 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:17:59.400572    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456" podUID="7ef5d329-0988-41ba-847d-34c0cd92d6be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Jul 08 17:18:00.608284 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:18:00.608231    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg" podUID="328715cc-4fc0-4fa6-9803-bfe419543da5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 08 17:18:09.387314 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:18:09.387270    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8001/health\": dial tcp 10.133.0.68:8001: connect: connection refused"
Jul 08 17:18:09.400690 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:18:09.400641    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456" podUID="7ef5d329-0988-41ba-847d-34c0cd92d6be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Jul 08 17:18:10.608921 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:18:10.608876    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg" podUID="328715cc-4fc0-4fa6-9803-bfe419543da5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 08 17:18:19.387075 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:18:19.387022    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8001/health\": dial tcp 10.133.0.68:8001: connect: connection refused"
Jul 08 17:18:19.401304 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:18:19.401263    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456" podUID="7ef5d329-0988-41ba-847d-34c0cd92d6be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Jul 08 17:18:20.609012 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:18:20.608966    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg" podUID="328715cc-4fc0-4fa6-9803-bfe419543da5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 08 17:18:29.387281 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:18:29.387234    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8001/health\": dial tcp 10.133.0.68:8001: connect: connection refused"
Jul 08 17:18:29.400367 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:18:29.400333    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456" podUID="7ef5d329-0988-41ba-847d-34c0cd92d6be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Jul 08 17:18:30.608543 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:18:30.608499    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg" podUID="328715cc-4fc0-4fa6-9803-bfe419543da5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 08 17:18:39.387061 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:18:39.387007    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8001/health\": dial tcp 10.133.0.68:8001: connect: connection refused"
Jul 08 17:18:39.401029 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:18:39.400991    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456" podUID="7ef5d329-0988-41ba-847d-34c0cd92d6be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Jul 08 17:18:40.608347 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:18:40.608307    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg" podUID="328715cc-4fc0-4fa6-9803-bfe419543da5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 08 17:18:49.387189 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:18:49.387136    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8001/health\": dial tcp 10.133.0.68:8001: connect: connection refused"
Jul 08 17:18:49.400530 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:18:49.400479    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456" podUID="7ef5d329-0988-41ba-847d-34c0cd92d6be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Jul 08 17:18:50.608716 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:18:50.608639    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg" podUID="328715cc-4fc0-4fa6-9803-bfe419543da5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 08 17:18:59.387357 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:18:59.387295    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8001/health\": dial tcp 10.133.0.68:8001: connect: connection refused"
Jul 08 17:18:59.400916 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:18:59.400870    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456" podUID="7ef5d329-0988-41ba-847d-34c0cd92d6be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Jul 08 17:19:00.609069 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:19:00.609021    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg" podUID="328715cc-4fc0-4fa6-9803-bfe419543da5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 08 17:19:09.387393 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:19:09.387347    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8001/health\": dial tcp 10.133.0.68:8001: connect: connection refused"
Jul 08 17:19:09.400464 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:19:09.400423    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456" podUID="7ef5d329-0988-41ba-847d-34c0cd92d6be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Jul 08 17:19:10.608469 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:19:10.608419    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg" podUID="328715cc-4fc0-4fa6-9803-bfe419543da5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 08 17:19:19.386980 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:19:19.386928    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8001/health\": dial tcp 10.133.0.68:8001: connect: connection refused"
Jul 08 17:19:19.400479 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:19:19.400439    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456" podUID="7ef5d329-0988-41ba-847d-34c0cd92d6be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Jul 08 17:19:20.609135 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:19:20.609086    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg" podUID="328715cc-4fc0-4fa6-9803-bfe419543da5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 08 17:19:29.387592 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:19:29.387529    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8001/health\": dial tcp 10.133.0.68:8001: connect: connection refused"
Jul 08 17:19:29.400628 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:19:29.400592    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456" podUID="7ef5d329-0988-41ba-847d-34c0cd92d6be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Jul 08 17:19:30.609002 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:19:30.608952    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg" podUID="328715cc-4fc0-4fa6-9803-bfe419543da5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 08 17:19:39.387498 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:19:39.387449    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8001/health\": dial tcp 10.133.0.68:8001: connect: connection refused"
Jul 08 17:19:39.400435 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:19:39.400395    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456" podUID="7ef5d329-0988-41ba-847d-34c0cd92d6be" containerName="main" probeResult="failure" output="Get \"https://10.133.0.69:8000/health\": dial tcp 10.133.0.69:8000: connect: connection refused"
Jul 08 17:19:40.608828 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:19:40.608775    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg" podUID="328715cc-4fc0-4fa6-9803-bfe419543da5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 08 17:19:49.397149 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:19:49.397113    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:19:49.409455 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:19:49.409420    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:19:49.411479 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:19:49.411449    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"
Jul 08 17:19:49.419271 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:19:49.419241    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"
Jul 08 17:19:50.609002 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:19:50.608948    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg" podUID="328715cc-4fc0-4fa6-9803-bfe419543da5" containerName="main" probeResult="failure" output="Get \"https://10.133.0.70:8000/health\": dial tcp 10.133.0.70:8000: connect: connection refused"
Jul 08 17:19:59.282823 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:19:59.282785    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"]
Jul 08 17:19:59.289181 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:19:59.289142    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"]
Jul 08 17:19:59.302287 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:19:59.302208    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:19:59.393887 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:19:59.393836    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:20:00.618363 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:00.618324    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"
Jul 08 17:20:00.626248 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:00.626218    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"
Jul 08 17:20:02.104500 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:02.104468    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"]
Jul 08 17:20:02.263481 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:02.263445    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:20:02.363765 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:02.363656    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:20:14.313468 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:14.313413    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456" podUID="7ef5d329-0988-41ba-847d-34c0cd92d6be" containerName="main" containerID="cri-o://77576122de76468cd563ff6dc9427b5c706fe92b8927ab9db132b560aae79b34" gracePeriod=45
Jul 08 17:20:14.316270 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:14.316225    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="main" containerID="cri-o://254ae93d4101a2d7cd85b3c57770bb28c13f7fa228d663209628f508012ebe8b" gracePeriod=45
Jul 08 17:20:15.126772 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.126739    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="llm-d-routing-sidecar" containerID="cri-o://d3cb180f19e4a58e872ad62f0c68ab9b7a1c5f50b88368b862e125ef161e3d8c" gracePeriod=45
Jul 08 17:20:15.279542 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.279513    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"
Jul 08 17:20:15.369340 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.369311    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:20:15.371240 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.371220    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-model-cache\") pod \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\" (UID: \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\") "
Jul 08 17:20:15.371302 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.371268    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbjfp\" (UniqueName: \"kubernetes.io/projected/51e1ec99-cae2-42e4-908b-c7bc01986b1e-kube-api-access-zbjfp\") pod \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\" (UID: \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\") "
Jul 08 17:20:15.371302 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.371298    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-tmp-dir\") pod \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\" (UID: \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\") "
Jul 08 17:20:15.371367 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.371319    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9c8d\" (UniqueName: \"kubernetes.io/projected/7ef5d329-0988-41ba-847d-34c0cd92d6be-kube-api-access-v9c8d\") pod \"7ef5d329-0988-41ba-847d-34c0cd92d6be\" (UID: \"7ef5d329-0988-41ba-847d-34c0cd92d6be\") "
Jul 08 17:20:15.371367 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.371342    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-home\") pod \"7ef5d329-0988-41ba-847d-34c0cd92d6be\" (UID: \"7ef5d329-0988-41ba-847d-34c0cd92d6be\") "
Jul 08 17:20:15.371523 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.371379    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-tmp-dir\") pod \"7ef5d329-0988-41ba-847d-34c0cd92d6be\" (UID: \"7ef5d329-0988-41ba-847d-34c0cd92d6be\") "
Jul 08 17:20:15.371523 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.371430    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-kserve-provision-location\") pod \"7ef5d329-0988-41ba-847d-34c0cd92d6be\" (UID: \"7ef5d329-0988-41ba-847d-34c0cd92d6be\") "
Jul 08 17:20:15.371523 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.371449    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-dshm\") pod \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\" (UID: \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\") "
Jul 08 17:20:15.371523 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.371477    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ef5d329-0988-41ba-847d-34c0cd92d6be-tls-certs\") pod \"7ef5d329-0988-41ba-847d-34c0cd92d6be\" (UID: \"7ef5d329-0988-41ba-847d-34c0cd92d6be\") "
Jul 08 17:20:15.371523 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.371488    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-model-cache" (OuterVolumeSpecName: "model-cache") pod "51e1ec99-cae2-42e4-908b-c7bc01986b1e" (UID: "51e1ec99-cae2-42e4-908b-c7bc01986b1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:20:15.371806 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.371510    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-kserve-provision-location\") pod \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\" (UID: \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\") "
Jul 08 17:20:15.371806 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.371570    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-model-cache\") pod \"7ef5d329-0988-41ba-847d-34c0cd92d6be\" (UID: \"7ef5d329-0988-41ba-847d-34c0cd92d6be\") "
Jul 08 17:20:15.371806 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.371603    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e1ec99-cae2-42e4-908b-c7bc01986b1e-tls-certs\") pod \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\" (UID: \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\") "
Jul 08 17:20:15.371806 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.371632    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-home\") pod \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\" (UID: \"51e1ec99-cae2-42e4-908b-c7bc01986b1e\") "
Jul 08 17:20:15.371806 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.371664    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-dshm\") pod \"7ef5d329-0988-41ba-847d-34c0cd92d6be\" (UID: \"7ef5d329-0988-41ba-847d-34c0cd92d6be\") "
Jul 08 17:20:15.372051 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.371992    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:20:15.372122 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.372099    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-home" (OuterVolumeSpecName: "home") pod "7ef5d329-0988-41ba-847d-34c0cd92d6be" (UID: "7ef5d329-0988-41ba-847d-34c0cd92d6be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:20:15.373103 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.373068    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-model-cache" (OuterVolumeSpecName: "model-cache") pod "7ef5d329-0988-41ba-847d-34c0cd92d6be" (UID: "7ef5d329-0988-41ba-847d-34c0cd92d6be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:20:15.374588 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.374565    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-home" (OuterVolumeSpecName: "home") pod "51e1ec99-cae2-42e4-908b-c7bc01986b1e" (UID: "51e1ec99-cae2-42e4-908b-c7bc01986b1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:20:15.374951 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.374929    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-dshm" (OuterVolumeSpecName: "dshm") pod "7ef5d329-0988-41ba-847d-34c0cd92d6be" (UID: "7ef5d329-0988-41ba-847d-34c0cd92d6be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:20:15.375306 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.375277    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51e1ec99-cae2-42e4-908b-c7bc01986b1e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "51e1ec99-cae2-42e4-908b-c7bc01986b1e" (UID: "51e1ec99-cae2-42e4-908b-c7bc01986b1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:20:15.375823 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.375786    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ef5d329-0988-41ba-847d-34c0cd92d6be-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7ef5d329-0988-41ba-847d-34c0cd92d6be" (UID: "7ef5d329-0988-41ba-847d-34c0cd92d6be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:20:15.376010 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.375983    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e1ec99-cae2-42e4-908b-c7bc01986b1e-kube-api-access-zbjfp" (OuterVolumeSpecName: "kube-api-access-zbjfp") pod "51e1ec99-cae2-42e4-908b-c7bc01986b1e" (UID: "51e1ec99-cae2-42e4-908b-c7bc01986b1e"). InnerVolumeSpecName "kube-api-access-zbjfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:20:15.377095 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.377062    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ef5d329-0988-41ba-847d-34c0cd92d6be-kube-api-access-v9c8d" (OuterVolumeSpecName: "kube-api-access-v9c8d") pod "7ef5d329-0988-41ba-847d-34c0cd92d6be" (UID: "7ef5d329-0988-41ba-847d-34c0cd92d6be"). InnerVolumeSpecName "kube-api-access-v9c8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:20:15.377874 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.377846    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-dshm" (OuterVolumeSpecName: "dshm") pod "51e1ec99-cae2-42e4-908b-c7bc01986b1e" (UID: "51e1ec99-cae2-42e4-908b-c7bc01986b1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:20:15.392124 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.392047    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7ef5d329-0988-41ba-847d-34c0cd92d6be" (UID: "7ef5d329-0988-41ba-847d-34c0cd92d6be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:20:15.392911 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.392882    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "51e1ec99-cae2-42e4-908b-c7bc01986b1e" (UID: "51e1ec99-cae2-42e4-908b-c7bc01986b1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:20:15.450879 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.450837    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7ef5d329-0988-41ba-847d-34c0cd92d6be" (UID: "7ef5d329-0988-41ba-847d-34c0cd92d6be"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:20:15.456825 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.456795    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "51e1ec99-cae2-42e4-908b-c7bc01986b1e" (UID: "51e1ec99-cae2-42e4-908b-c7bc01986b1e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:20:15.472960 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.472926    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-kserve-provision-location\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:20:15.472960 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.472959    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:20:15.473130 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.472969    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ef5d329-0988-41ba-847d-34c0cd92d6be-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:20:15.473130 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.472980    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-kserve-provision-location\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:20:15.473130 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.472988    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:20:15.473130 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.472996    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51e1ec99-cae2-42e4-908b-c7bc01986b1e-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:20:15.473130 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.473004    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:20:15.473130 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.473012    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:20:15.473130 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.473020    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbjfp\" (UniqueName: \"kubernetes.io/projected/51e1ec99-cae2-42e4-908b-c7bc01986b1e-kube-api-access-zbjfp\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:20:15.473130 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.473028    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51e1ec99-cae2-42e4-908b-c7bc01986b1e-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:20:15.473130 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.473036    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9c8d\" (UniqueName: \"kubernetes.io/projected/7ef5d329-0988-41ba-847d-34c0cd92d6be-kube-api-access-v9c8d\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:20:15.473130 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.473044    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:20:15.473130 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.473051    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ef5d329-0988-41ba-847d-34c0cd92d6be-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:20:15.595190 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.595098    2565 generic.go:356] "Generic (PLEG): container finished" podID="7ef5d329-0988-41ba-847d-34c0cd92d6be" containerID="77576122de76468cd563ff6dc9427b5c706fe92b8927ab9db132b560aae79b34" exitCode=0
Jul 08 17:20:15.595190 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.595131    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456" event={"ID":"7ef5d329-0988-41ba-847d-34c0cd92d6be","Type":"ContainerDied","Data":"77576122de76468cd563ff6dc9427b5c706fe92b8927ab9db132b560aae79b34"}
Jul 08 17:20:15.595190 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.595162    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"
Jul 08 17:20:15.595456 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.595222    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456" event={"ID":"7ef5d329-0988-41ba-847d-34c0cd92d6be","Type":"ContainerDied","Data":"5b976265fb1b046bd050c98d730c920e65713266e6038a46e3f6011a74ecc78b"}
Jul 08 17:20:15.595456 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.595247    2565 scope.go:117] "RemoveContainer" containerID="77576122de76468cd563ff6dc9427b5c706fe92b8927ab9db132b560aae79b34"
Jul 08 17:20:15.597228 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.597197    2565 generic.go:356] "Generic (PLEG): container finished" podID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerID="254ae93d4101a2d7cd85b3c57770bb28c13f7fa228d663209628f508012ebe8b" exitCode=0
Jul 08 17:20:15.597344 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.597229    2565 generic.go:356] "Generic (PLEG): container finished" podID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerID="d3cb180f19e4a58e872ad62f0c68ab9b7a1c5f50b88368b862e125ef161e3d8c" exitCode=0
Jul 08 17:20:15.597344 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.597260    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"
Jul 08 17:20:15.597344 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.597275    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" event={"ID":"51e1ec99-cae2-42e4-908b-c7bc01986b1e","Type":"ContainerDied","Data":"254ae93d4101a2d7cd85b3c57770bb28c13f7fa228d663209628f508012ebe8b"}
Jul 08 17:20:15.597344 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.597339    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" event={"ID":"51e1ec99-cae2-42e4-908b-c7bc01986b1e","Type":"ContainerDied","Data":"d3cb180f19e4a58e872ad62f0c68ab9b7a1c5f50b88368b862e125ef161e3d8c"}
Jul 08 17:20:15.597484 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.597372    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t" event={"ID":"51e1ec99-cae2-42e4-908b-c7bc01986b1e","Type":"ContainerDied","Data":"a23383da6c3d9d565fabe17f52debea223ab3499eceffebec33e0b82f75e4af7"}
Jul 08 17:20:15.604816 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.604796    2565 scope.go:117] "RemoveContainer" containerID="82975f5e225cca6c6e6dd7178887fa53abb523b4ac69425f051e5b90f6264a86"
Jul 08 17:20:15.614463 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.614438    2565 scope.go:117] "RemoveContainer" containerID="77576122de76468cd563ff6dc9427b5c706fe92b8927ab9db132b560aae79b34"
Jul 08 17:20:15.614768 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:20:15.614748    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77576122de76468cd563ff6dc9427b5c706fe92b8927ab9db132b560aae79b34\": container with ID starting with 77576122de76468cd563ff6dc9427b5c706fe92b8927ab9db132b560aae79b34 not found: ID does not exist" containerID="77576122de76468cd563ff6dc9427b5c706fe92b8927ab9db132b560aae79b34"
Jul 08 17:20:15.614824 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.614777    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77576122de76468cd563ff6dc9427b5c706fe92b8927ab9db132b560aae79b34"} err="failed to get container status \"77576122de76468cd563ff6dc9427b5c706fe92b8927ab9db132b560aae79b34\": rpc error: code = NotFound desc = could not find container \"77576122de76468cd563ff6dc9427b5c706fe92b8927ab9db132b560aae79b34\": container with ID starting with 77576122de76468cd563ff6dc9427b5c706fe92b8927ab9db132b560aae79b34 not found: ID does not exist"
Jul 08 17:20:15.614824 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.614795    2565 scope.go:117] "RemoveContainer" containerID="82975f5e225cca6c6e6dd7178887fa53abb523b4ac69425f051e5b90f6264a86"
Jul 08 17:20:15.615044 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:20:15.615024    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82975f5e225cca6c6e6dd7178887fa53abb523b4ac69425f051e5b90f6264a86\": container with ID starting with 82975f5e225cca6c6e6dd7178887fa53abb523b4ac69425f051e5b90f6264a86 not found: ID does not exist" containerID="82975f5e225cca6c6e6dd7178887fa53abb523b4ac69425f051e5b90f6264a86"
Jul 08 17:20:15.615085 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.615051    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82975f5e225cca6c6e6dd7178887fa53abb523b4ac69425f051e5b90f6264a86"} err="failed to get container status \"82975f5e225cca6c6e6dd7178887fa53abb523b4ac69425f051e5b90f6264a86\": rpc error: code = NotFound desc = could not find container \"82975f5e225cca6c6e6dd7178887fa53abb523b4ac69425f051e5b90f6264a86\": container with ID starting with 82975f5e225cca6c6e6dd7178887fa53abb523b4ac69425f051e5b90f6264a86 not found: ID does not exist"
Jul 08 17:20:15.615085 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.615066    2565 scope.go:117] "RemoveContainer" containerID="254ae93d4101a2d7cd85b3c57770bb28c13f7fa228d663209628f508012ebe8b"
Jul 08 17:20:15.621571 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.621550    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"]
Jul 08 17:20:15.624867 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.624849    2565 scope.go:117] "RemoveContainer" containerID="a74a774bc30c381a4c1a970ec4ffa111652c602d21116b4d641e4ad02b38ffab"
Jul 08 17:20:15.630861 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.630839    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-prefill-94b6f6947-rx456"]
Jul 08 17:20:15.636214 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.636192    2565 scope.go:117] "RemoveContainer" containerID="d3cb180f19e4a58e872ad62f0c68ab9b7a1c5f50b88368b862e125ef161e3d8c"
Jul 08 17:20:15.644497 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.644464    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"]
Jul 08 17:20:15.647796 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.647697    2565 scope.go:117] "RemoveContainer" containerID="254ae93d4101a2d7cd85b3c57770bb28c13f7fa228d663209628f508012ebe8b"
Jul 08 17:20:15.648109 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:20:15.648081    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"254ae93d4101a2d7cd85b3c57770bb28c13f7fa228d663209628f508012ebe8b\": container with ID starting with 254ae93d4101a2d7cd85b3c57770bb28c13f7fa228d663209628f508012ebe8b not found: ID does not exist" containerID="254ae93d4101a2d7cd85b3c57770bb28c13f7fa228d663209628f508012ebe8b"
Jul 08 17:20:15.648223 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.648113    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"254ae93d4101a2d7cd85b3c57770bb28c13f7fa228d663209628f508012ebe8b"} err="failed to get container status \"254ae93d4101a2d7cd85b3c57770bb28c13f7fa228d663209628f508012ebe8b\": rpc error: code = NotFound desc = could not find container \"254ae93d4101a2d7cd85b3c57770bb28c13f7fa228d663209628f508012ebe8b\": container with ID starting with 254ae93d4101a2d7cd85b3c57770bb28c13f7fa228d663209628f508012ebe8b not found: ID does not exist"
Jul 08 17:20:15.648223 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.648131    2565 scope.go:117] "RemoveContainer" containerID="a74a774bc30c381a4c1a970ec4ffa111652c602d21116b4d641e4ad02b38ffab"
Jul 08 17:20:15.648473 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:20:15.648443    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a74a774bc30c381a4c1a970ec4ffa111652c602d21116b4d641e4ad02b38ffab\": container with ID starting with a74a774bc30c381a4c1a970ec4ffa111652c602d21116b4d641e4ad02b38ffab not found: ID does not exist" containerID="a74a774bc30c381a4c1a970ec4ffa111652c602d21116b4d641e4ad02b38ffab"
Jul 08 17:20:15.648542 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.648471    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a74a774bc30c381a4c1a970ec4ffa111652c602d21116b4d641e4ad02b38ffab"} err="failed to get container status \"a74a774bc30c381a4c1a970ec4ffa111652c602d21116b4d641e4ad02b38ffab\": rpc error: code = NotFound desc = could not find container \"a74a774bc30c381a4c1a970ec4ffa111652c602d21116b4d641e4ad02b38ffab\": container with ID starting with a74a774bc30c381a4c1a970ec4ffa111652c602d21116b4d641e4ad02b38ffab not found: ID does not exist"
Jul 08 17:20:15.648542 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.648488    2565 scope.go:117] "RemoveContainer" containerID="d3cb180f19e4a58e872ad62f0c68ab9b7a1c5f50b88368b862e125ef161e3d8c"
Jul 08 17:20:15.648971 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:20:15.648951    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3cb180f19e4a58e872ad62f0c68ab9b7a1c5f50b88368b862e125ef161e3d8c\": container with ID starting with d3cb180f19e4a58e872ad62f0c68ab9b7a1c5f50b88368b862e125ef161e3d8c not found: ID does not exist" containerID="d3cb180f19e4a58e872ad62f0c68ab9b7a1c5f50b88368b862e125ef161e3d8c"
Jul 08 17:20:15.649072 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.648974    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3cb180f19e4a58e872ad62f0c68ab9b7a1c5f50b88368b862e125ef161e3d8c"} err="failed to get container status \"d3cb180f19e4a58e872ad62f0c68ab9b7a1c5f50b88368b862e125ef161e3d8c\": rpc error: code = NotFound desc = could not find container \"d3cb180f19e4a58e872ad62f0c68ab9b7a1c5f50b88368b862e125ef161e3d8c\": container with ID starting with d3cb180f19e4a58e872ad62f0c68ab9b7a1c5f50b88368b862e125ef161e3d8c not found: ID does not exist"
Jul 08 17:20:15.649072 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.648989    2565 scope.go:117] "RemoveContainer" containerID="254ae93d4101a2d7cd85b3c57770bb28c13f7fa228d663209628f508012ebe8b"
Jul 08 17:20:15.649270 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.649243    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"254ae93d4101a2d7cd85b3c57770bb28c13f7fa228d663209628f508012ebe8b"} err="failed to get container status \"254ae93d4101a2d7cd85b3c57770bb28c13f7fa228d663209628f508012ebe8b\": rpc error: code = NotFound desc = could not find container \"254ae93d4101a2d7cd85b3c57770bb28c13f7fa228d663209628f508012ebe8b\": container with ID starting with 254ae93d4101a2d7cd85b3c57770bb28c13f7fa228d663209628f508012ebe8b not found: ID does not exist"
Jul 08 17:20:15.649346 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.649272    2565 scope.go:117] "RemoveContainer" containerID="a74a774bc30c381a4c1a970ec4ffa111652c602d21116b4d641e4ad02b38ffab"
Jul 08 17:20:15.649542 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.649523    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a74a774bc30c381a4c1a970ec4ffa111652c602d21116b4d641e4ad02b38ffab"} err="failed to get container status \"a74a774bc30c381a4c1a970ec4ffa111652c602d21116b4d641e4ad02b38ffab\": rpc error: code = NotFound desc = could not find container \"a74a774bc30c381a4c1a970ec4ffa111652c602d21116b4d641e4ad02b38ffab\": container with ID starting with a74a774bc30c381a4c1a970ec4ffa111652c602d21116b4d641e4ad02b38ffab not found: ID does not exist"
Jul 08 17:20:15.649619 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.649543    2565 scope.go:117] "RemoveContainer" containerID="d3cb180f19e4a58e872ad62f0c68ab9b7a1c5f50b88368b862e125ef161e3d8c"
Jul 08 17:20:15.649853 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.649831    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3cb180f19e4a58e872ad62f0c68ab9b7a1c5f50b88368b862e125ef161e3d8c"} err="failed to get container status \"d3cb180f19e4a58e872ad62f0c68ab9b7a1c5f50b88368b862e125ef161e3d8c\": rpc error: code = NotFound desc = could not find container \"d3cb180f19e4a58e872ad62f0c68ab9b7a1c5f50b88368b862e125ef161e3d8c\": container with ID starting with d3cb180f19e4a58e872ad62f0c68ab9b7a1c5f50b88368b862e125ef161e3d8c not found: ID does not exist"
Jul 08 17:20:15.649853 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.649839    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/custom-route-timeout-pd-test-kserve-67588db874-f6b8t"]
Jul 08 17:20:15.822899 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.822854    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" path="/var/lib/kubelet/pods/51e1ec99-cae2-42e4-908b-c7bc01986b1e/volumes"
Jul 08 17:20:15.823452 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:15.823431    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ef5d329-0988-41ba-847d-34c0cd92d6be" path="/var/lib/kubelet/pods/7ef5d329-0988-41ba-847d-34c0cd92d6be/volumes"
Jul 08 17:20:17.545470 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:17.545425    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg" podUID="328715cc-4fc0-4fa6-9803-bfe419543da5" containerName="main" containerID="cri-o://d376691e08b9024b6118a8fffdba1a11e8b9e2f5bc0ade8b475f3185c7b655c9" gracePeriod=45
Jul 08 17:20:18.223853 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.223830    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"
Jul 08 17:20:18.296306 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.296212    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-tmp-dir\") pod \"328715cc-4fc0-4fa6-9803-bfe419543da5\" (UID: \"328715cc-4fc0-4fa6-9803-bfe419543da5\") "
Jul 08 17:20:18.296306 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.296265    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/328715cc-4fc0-4fa6-9803-bfe419543da5-tls-certs\") pod \"328715cc-4fc0-4fa6-9803-bfe419543da5\" (UID: \"328715cc-4fc0-4fa6-9803-bfe419543da5\") "
Jul 08 17:20:18.296514 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.296318    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-dshm\") pod \"328715cc-4fc0-4fa6-9803-bfe419543da5\" (UID: \"328715cc-4fc0-4fa6-9803-bfe419543da5\") "
Jul 08 17:20:18.296514 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.296334    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-model-cache\") pod \"328715cc-4fc0-4fa6-9803-bfe419543da5\" (UID: \"328715cc-4fc0-4fa6-9803-bfe419543da5\") "
Jul 08 17:20:18.296514 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.296367    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4r9bs\" (UniqueName: \"kubernetes.io/projected/328715cc-4fc0-4fa6-9803-bfe419543da5-kube-api-access-4r9bs\") pod \"328715cc-4fc0-4fa6-9803-bfe419543da5\" (UID: \"328715cc-4fc0-4fa6-9803-bfe419543da5\") "
Jul 08 17:20:18.296514 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.296389    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-kserve-provision-location\") pod \"328715cc-4fc0-4fa6-9803-bfe419543da5\" (UID: \"328715cc-4fc0-4fa6-9803-bfe419543da5\") "
Jul 08 17:20:18.296514 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.296416    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-home\") pod \"328715cc-4fc0-4fa6-9803-bfe419543da5\" (UID: \"328715cc-4fc0-4fa6-9803-bfe419543da5\") "
Jul 08 17:20:18.296812 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.296667    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-model-cache" (OuterVolumeSpecName: "model-cache") pod "328715cc-4fc0-4fa6-9803-bfe419543da5" (UID: "328715cc-4fc0-4fa6-9803-bfe419543da5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:20:18.297216 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.297189    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-home" (OuterVolumeSpecName: "home") pod "328715cc-4fc0-4fa6-9803-bfe419543da5" (UID: "328715cc-4fc0-4fa6-9803-bfe419543da5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:20:18.298570 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.298548    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/328715cc-4fc0-4fa6-9803-bfe419543da5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "328715cc-4fc0-4fa6-9803-bfe419543da5" (UID: "328715cc-4fc0-4fa6-9803-bfe419543da5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:20:18.298887 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.298864    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-dshm" (OuterVolumeSpecName: "dshm") pod "328715cc-4fc0-4fa6-9803-bfe419543da5" (UID: "328715cc-4fc0-4fa6-9803-bfe419543da5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:20:18.299076 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.299046    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/328715cc-4fc0-4fa6-9803-bfe419543da5-kube-api-access-4r9bs" (OuterVolumeSpecName: "kube-api-access-4r9bs") pod "328715cc-4fc0-4fa6-9803-bfe419543da5" (UID: "328715cc-4fc0-4fa6-9803-bfe419543da5"). InnerVolumeSpecName "kube-api-access-4r9bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:20:18.310512 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.310480    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "328715cc-4fc0-4fa6-9803-bfe419543da5" (UID: "328715cc-4fc0-4fa6-9803-bfe419543da5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:20:18.356535 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.356497    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "328715cc-4fc0-4fa6-9803-bfe419543da5" (UID: "328715cc-4fc0-4fa6-9803-bfe419543da5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:20:18.397241 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.397202    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:20:18.397241 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.397235    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:20:18.397241 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.397248    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r9bs\" (UniqueName: \"kubernetes.io/projected/328715cc-4fc0-4fa6-9803-bfe419543da5-kube-api-access-4r9bs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:20:18.397472 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.397258    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-kserve-provision-location\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:20:18.397472 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.397268    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:20:18.397472 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.397276    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/328715cc-4fc0-4fa6-9803-bfe419543da5-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:20:18.397472 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.397283    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/328715cc-4fc0-4fa6-9803-bfe419543da5-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:20:18.617036 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.616946    2565 generic.go:356] "Generic (PLEG): container finished" podID="328715cc-4fc0-4fa6-9803-bfe419543da5" containerID="d376691e08b9024b6118a8fffdba1a11e8b9e2f5bc0ade8b475f3185c7b655c9" exitCode=0
Jul 08 17:20:18.617036 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.616995    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"
Jul 08 17:20:18.617036 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.617018    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg" event={"ID":"328715cc-4fc0-4fa6-9803-bfe419543da5","Type":"ContainerDied","Data":"d376691e08b9024b6118a8fffdba1a11e8b9e2f5bc0ade8b475f3185c7b655c9"}
Jul 08 17:20:18.617590 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.617069    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg" event={"ID":"328715cc-4fc0-4fa6-9803-bfe419543da5","Type":"ContainerDied","Data":"014d135232e6eef94296f9697a9741e0e85a21cf1f51c26f18cd2d7281f83330"}
Jul 08 17:20:18.617590 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.617093    2565 scope.go:117] "RemoveContainer" containerID="d376691e08b9024b6118a8fffdba1a11e8b9e2f5bc0ade8b475f3185c7b655c9"
Jul 08 17:20:18.627313 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.627291    2565 scope.go:117] "RemoveContainer" containerID="70ce2fc7fd56ffbd591150fcf9934ce922fdcfe8652aa9c97122987290e2f00f"
Jul 08 17:20:18.636633 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.636608    2565 scope.go:117] "RemoveContainer" containerID="d376691e08b9024b6118a8fffdba1a11e8b9e2f5bc0ade8b475f3185c7b655c9"
Jul 08 17:20:18.636999 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:20:18.636969    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d376691e08b9024b6118a8fffdba1a11e8b9e2f5bc0ade8b475f3185c7b655c9\": container with ID starting with d376691e08b9024b6118a8fffdba1a11e8b9e2f5bc0ade8b475f3185c7b655c9 not found: ID does not exist" containerID="d376691e08b9024b6118a8fffdba1a11e8b9e2f5bc0ade8b475f3185c7b655c9"
Jul 08 17:20:18.637112 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.637004    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d376691e08b9024b6118a8fffdba1a11e8b9e2f5bc0ade8b475f3185c7b655c9"} err="failed to get container status \"d376691e08b9024b6118a8fffdba1a11e8b9e2f5bc0ade8b475f3185c7b655c9\": rpc error: code = NotFound desc = could not find container \"d376691e08b9024b6118a8fffdba1a11e8b9e2f5bc0ade8b475f3185c7b655c9\": container with ID starting with d376691e08b9024b6118a8fffdba1a11e8b9e2f5bc0ade8b475f3185c7b655c9 not found: ID does not exist"
Jul 08 17:20:18.637112 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.637023    2565 scope.go:117] "RemoveContainer" containerID="70ce2fc7fd56ffbd591150fcf9934ce922fdcfe8652aa9c97122987290e2f00f"
Jul 08 17:20:18.637291 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:20:18.637269    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70ce2fc7fd56ffbd591150fcf9934ce922fdcfe8652aa9c97122987290e2f00f\": container with ID starting with 70ce2fc7fd56ffbd591150fcf9934ce922fdcfe8652aa9c97122987290e2f00f not found: ID does not exist" containerID="70ce2fc7fd56ffbd591150fcf9934ce922fdcfe8652aa9c97122987290e2f00f"
Jul 08 17:20:18.637333 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.637298    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70ce2fc7fd56ffbd591150fcf9934ce922fdcfe8652aa9c97122987290e2f00f"} err="failed to get container status \"70ce2fc7fd56ffbd591150fcf9934ce922fdcfe8652aa9c97122987290e2f00f\": rpc error: code = NotFound desc = could not find container \"70ce2fc7fd56ffbd591150fcf9934ce922fdcfe8652aa9c97122987290e2f00f\": container with ID starting with 70ce2fc7fd56ffbd591150fcf9934ce922fdcfe8652aa9c97122987290e2f00f not found: ID does not exist"
Jul 08 17:20:18.663698 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.647505    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"]
Jul 08 17:20:18.672812 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:18.672766    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/stop-feature-test-kserve-7c7845f79f-tfvwg"]
Jul 08 17:20:19.827838 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:19.827803    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="328715cc-4fc0-4fa6-9803-bfe419543da5" path="/var/lib/kubelet/pods/328715cc-4fc0-4fa6-9803-bfe419543da5/volumes"
Jul 08 17:20:20.182644 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.182611    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:20:20.282621 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.282580    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:20:20.464939 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.464898    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"]
Jul 08 17:20:20.465331 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.465315    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="main"
Jul 08 17:20:20.465331 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.465332    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="main"
Jul 08 17:20:20.465436 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.465343    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="storage-initializer"
Jul 08 17:20:20.465436 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.465349    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="storage-initializer"
Jul 08 17:20:20.465436 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.465362    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="328715cc-4fc0-4fa6-9803-bfe419543da5" containerName="main"
Jul 08 17:20:20.465436 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.465367    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="328715cc-4fc0-4fa6-9803-bfe419543da5" containerName="main"
Jul 08 17:20:20.465436 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.465376    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ef5d329-0988-41ba-847d-34c0cd92d6be" containerName="storage-initializer"
Jul 08 17:20:20.465436 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.465382    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ef5d329-0988-41ba-847d-34c0cd92d6be" containerName="storage-initializer"
Jul 08 17:20:20.465436 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.465388    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="llm-d-routing-sidecar"
Jul 08 17:20:20.465436 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.465394    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="llm-d-routing-sidecar"
Jul 08 17:20:20.465436 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.465399    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="328715cc-4fc0-4fa6-9803-bfe419543da5" containerName="storage-initializer"
Jul 08 17:20:20.465436 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.465404    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="328715cc-4fc0-4fa6-9803-bfe419543da5" containerName="storage-initializer"
Jul 08 17:20:20.465436 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.465411    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ef5d329-0988-41ba-847d-34c0cd92d6be" containerName="main"
Jul 08 17:20:20.465436 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.465418    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ef5d329-0988-41ba-847d-34c0cd92d6be" containerName="main"
Jul 08 17:20:20.465850 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.465480    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="llm-d-routing-sidecar"
Jul 08 17:20:20.465850 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.465489    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ef5d329-0988-41ba-847d-34c0cd92d6be" containerName="main"
Jul 08 17:20:20.465850 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.465494    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="328715cc-4fc0-4fa6-9803-bfe419543da5" containerName="main"
Jul 08 17:20:20.465850 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.465500    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="51e1ec99-cae2-42e4-908b-c7bc01986b1e" containerName="main"
Jul 08 17:20:20.468826 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.468804    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:20.471150 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.471129    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-gateway-2-openshift-default-dockercfg-kvqnt\""
Jul 08 17:20:20.481945 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.481919    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"]
Jul 08 17:20:20.517647 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.517599    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49d08e4c-b706-4071-a94c-26aca5a1fa9a-istio-data\") pod \"router-gateway-2-openshift-default-684dc9b4b9-h2x2d\" (UID: \"49d08e4c-b706-4071-a94c-26aca5a1fa9a\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:20.517859 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.517659    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d08e4c-b706-4071-a94c-26aca5a1fa9a-credential-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-h2x2d\" (UID: \"49d08e4c-b706-4071-a94c-26aca5a1fa9a\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:20.517859 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.517727    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49d08e4c-b706-4071-a94c-26aca5a1fa9a-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-684dc9b4b9-h2x2d\" (UID: \"49d08e4c-b706-4071-a94c-26aca5a1fa9a\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:20.517859 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.517754    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49d08e4c-b706-4071-a94c-26aca5a1fa9a-istio-envoy\") pod \"router-gateway-2-openshift-default-684dc9b4b9-h2x2d\" (UID: \"49d08e4c-b706-4071-a94c-26aca5a1fa9a\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:20.517971 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.517852    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d08e4c-b706-4071-a94c-26aca5a1fa9a-workload-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-h2x2d\" (UID: \"49d08e4c-b706-4071-a94c-26aca5a1fa9a\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:20.517971 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.517887    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49d08e4c-b706-4071-a94c-26aca5a1fa9a-istio-podinfo\") pod \"router-gateway-2-openshift-default-684dc9b4b9-h2x2d\" (UID: \"49d08e4c-b706-4071-a94c-26aca5a1fa9a\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:20.517971 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.517906    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sgtt\" (UniqueName: \"kubernetes.io/projected/49d08e4c-b706-4071-a94c-26aca5a1fa9a-kube-api-access-8sgtt\") pod \"router-gateway-2-openshift-default-684dc9b4b9-h2x2d\" (UID: \"49d08e4c-b706-4071-a94c-26aca5a1fa9a\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:20.518070 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.517994    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49d08e4c-b706-4071-a94c-26aca5a1fa9a-istio-token\") pod \"router-gateway-2-openshift-default-684dc9b4b9-h2x2d\" (UID: \"49d08e4c-b706-4071-a94c-26aca5a1fa9a\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:20.518070 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.518032    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49d08e4c-b706-4071-a94c-26aca5a1fa9a-workload-certs\") pod \"router-gateway-2-openshift-default-684dc9b4b9-h2x2d\" (UID: \"49d08e4c-b706-4071-a94c-26aca5a1fa9a\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:20.619404 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.619373    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49d08e4c-b706-4071-a94c-26aca5a1fa9a-istio-envoy\") pod \"router-gateway-2-openshift-default-684dc9b4b9-h2x2d\" (UID: \"49d08e4c-b706-4071-a94c-26aca5a1fa9a\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:20.619610 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.619417    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d08e4c-b706-4071-a94c-26aca5a1fa9a-workload-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-h2x2d\" (UID: \"49d08e4c-b706-4071-a94c-26aca5a1fa9a\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:20.619610 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.619435    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49d08e4c-b706-4071-a94c-26aca5a1fa9a-istio-podinfo\") pod \"router-gateway-2-openshift-default-684dc9b4b9-h2x2d\" (UID: \"49d08e4c-b706-4071-a94c-26aca5a1fa9a\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:20.619610 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.619459    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sgtt\" (UniqueName: \"kubernetes.io/projected/49d08e4c-b706-4071-a94c-26aca5a1fa9a-kube-api-access-8sgtt\") pod \"router-gateway-2-openshift-default-684dc9b4b9-h2x2d\" (UID: \"49d08e4c-b706-4071-a94c-26aca5a1fa9a\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:20.619610 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.619508    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49d08e4c-b706-4071-a94c-26aca5a1fa9a-istio-token\") pod \"router-gateway-2-openshift-default-684dc9b4b9-h2x2d\" (UID: \"49d08e4c-b706-4071-a94c-26aca5a1fa9a\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:20.619610 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.619534    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49d08e4c-b706-4071-a94c-26aca5a1fa9a-workload-certs\") pod \"router-gateway-2-openshift-default-684dc9b4b9-h2x2d\" (UID: \"49d08e4c-b706-4071-a94c-26aca5a1fa9a\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:20.619927 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.619642    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49d08e4c-b706-4071-a94c-26aca5a1fa9a-istio-data\") pod \"router-gateway-2-openshift-default-684dc9b4b9-h2x2d\" (UID: \"49d08e4c-b706-4071-a94c-26aca5a1fa9a\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:20.619927 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.619727    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d08e4c-b706-4071-a94c-26aca5a1fa9a-credential-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-h2x2d\" (UID: \"49d08e4c-b706-4071-a94c-26aca5a1fa9a\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:20.619927 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.619785    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49d08e4c-b706-4071-a94c-26aca5a1fa9a-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-684dc9b4b9-h2x2d\" (UID: \"49d08e4c-b706-4071-a94c-26aca5a1fa9a\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:20.619927 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.619921    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d08e4c-b706-4071-a94c-26aca5a1fa9a-workload-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-h2x2d\" (UID: \"49d08e4c-b706-4071-a94c-26aca5a1fa9a\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:20.620134 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.619959    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/49d08e4c-b706-4071-a94c-26aca5a1fa9a-istio-data\") pod \"router-gateway-2-openshift-default-684dc9b4b9-h2x2d\" (UID: \"49d08e4c-b706-4071-a94c-26aca5a1fa9a\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:20.620134 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.620111    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/49d08e4c-b706-4071-a94c-26aca5a1fa9a-workload-certs\") pod \"router-gateway-2-openshift-default-684dc9b4b9-h2x2d\" (UID: \"49d08e4c-b706-4071-a94c-26aca5a1fa9a\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:20.620229 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.620143    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/49d08e4c-b706-4071-a94c-26aca5a1fa9a-credential-socket\") pod \"router-gateway-2-openshift-default-684dc9b4b9-h2x2d\" (UID: \"49d08e4c-b706-4071-a94c-26aca5a1fa9a\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:20.620545 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.620507    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/49d08e4c-b706-4071-a94c-26aca5a1fa9a-istiod-ca-cert\") pod \"router-gateway-2-openshift-default-684dc9b4b9-h2x2d\" (UID: \"49d08e4c-b706-4071-a94c-26aca5a1fa9a\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:20.621901 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.621883    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/49d08e4c-b706-4071-a94c-26aca5a1fa9a-istio-envoy\") pod \"router-gateway-2-openshift-default-684dc9b4b9-h2x2d\" (UID: \"49d08e4c-b706-4071-a94c-26aca5a1fa9a\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:20.622561 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.622538    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/49d08e4c-b706-4071-a94c-26aca5a1fa9a-istio-podinfo\") pod \"router-gateway-2-openshift-default-684dc9b4b9-h2x2d\" (UID: \"49d08e4c-b706-4071-a94c-26aca5a1fa9a\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:20.628526 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.628505    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sgtt\" (UniqueName: \"kubernetes.io/projected/49d08e4c-b706-4071-a94c-26aca5a1fa9a-kube-api-access-8sgtt\") pod \"router-gateway-2-openshift-default-684dc9b4b9-h2x2d\" (UID: \"49d08e4c-b706-4071-a94c-26aca5a1fa9a\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:20.628727 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.628706    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/49d08e4c-b706-4071-a94c-26aca5a1fa9a-istio-token\") pod \"router-gateway-2-openshift-default-684dc9b4b9-h2x2d\" (UID: \"49d08e4c-b706-4071-a94c-26aca5a1fa9a\") " pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:20.780597 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.780510    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:20.927393 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.927371    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"]
Jul 08 17:20:20.929386 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:20:20.929356    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d08e4c_b706_4071_a94c_26aca5a1fa9a.slice/crio-f991d32439a6a9d6266f50397a954ccc602259c08a0def2fb5e7c41808f2efdd WatchSource:0}: Error finding container f991d32439a6a9d6266f50397a954ccc602259c08a0def2fb5e7c41808f2efdd: Status 404 returned error can't find the container with id f991d32439a6a9d6266f50397a954ccc602259c08a0def2fb5e7c41808f2efdd
Jul 08 17:20:20.931859 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.931822    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 08 17:20:20.931953 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.931890    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 08 17:20:20.931953 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:20.931917    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 08 17:20:21.636865 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:21.636828    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d" event={"ID":"49d08e4c-b706-4071-a94c-26aca5a1fa9a","Type":"ContainerStarted","Data":"fe495a985461ffa05a10f7430014a4e7769be9d4927f8b342f311fedfca9c2a1"}
Jul 08 17:20:21.637146 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:21.636885    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d" event={"ID":"49d08e4c-b706-4071-a94c-26aca5a1fa9a","Type":"ContainerStarted","Data":"f991d32439a6a9d6266f50397a954ccc602259c08a0def2fb5e7c41808f2efdd"}
Jul 08 17:20:21.662765 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:21.662711    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d" podStartSLOduration=1.6626924239999998 podStartE2EDuration="1.662692424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:20:21.659961984 +0000 UTC m=+2758.373323432" watchObservedRunningTime="2026-07-08 17:20:21.662692424 +0000 UTC m=+2758.376053841"
Jul 08 17:20:21.780847 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:21.780810    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:21.785830 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:21.785802    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:22.642468 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:22.642436    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:22.643514 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:22.643494    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:23.797346 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:23.797306    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:20:29.763001 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.762967    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"]
Jul 08 17:20:29.766413 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.766394    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:20:29.769066 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.769034    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 08 17:20:29.769209 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.769079    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"lora-single-adapter-test-kserve-self-signed-certs\""
Jul 08 17:20:29.769349 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.769334    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qxdr\""
Jul 08 17:20:29.769420 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.769364    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 08 17:20:29.781045 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.781016    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"]
Jul 08 17:20:29.789811 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.789781    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:20:29.807404 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.807375    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-home\") pod \"lora-single-adapter-test-kserve-75fdddbc5d-r8z7x\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:20:29.807533 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.807418    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4742f196-ebc8-4851-bad6-ab27dab5f5ac-cabundle-cert\") pod \"lora-single-adapter-test-kserve-75fdddbc5d-r8z7x\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:20:29.807533 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.807477    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-75fdddbc5d-r8z7x\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:20:29.807533 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.807496    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-model-cache\") pod \"lora-single-adapter-test-kserve-75fdddbc5d-r8z7x\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:20:29.807533 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.807526    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4742f196-ebc8-4851-bad6-ab27dab5f5ac-tls-certs\") pod \"lora-single-adapter-test-kserve-75fdddbc5d-r8z7x\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:20:29.807709 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.807555    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k6s6\" (UniqueName: \"kubernetes.io/projected/4742f196-ebc8-4851-bad6-ab27dab5f5ac-kube-api-access-2k6s6\") pod \"lora-single-adapter-test-kserve-75fdddbc5d-r8z7x\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:20:29.807709 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.807576    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-dshm\") pod \"lora-single-adapter-test-kserve-75fdddbc5d-r8z7x\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:20:29.807709 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.807595    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-tmp-dir\") pod \"lora-single-adapter-test-kserve-75fdddbc5d-r8z7x\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:20:29.909070 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.909037    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-tmp-dir\") pod \"lora-single-adapter-test-kserve-75fdddbc5d-r8z7x\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:20:29.909240 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.909088    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-home\") pod \"lora-single-adapter-test-kserve-75fdddbc5d-r8z7x\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:20:29.909240 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.909120    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4742f196-ebc8-4851-bad6-ab27dab5f5ac-cabundle-cert\") pod \"lora-single-adapter-test-kserve-75fdddbc5d-r8z7x\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:20:29.909240 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.909171    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-75fdddbc5d-r8z7x\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:20:29.909240 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.909198    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-model-cache\") pod \"lora-single-adapter-test-kserve-75fdddbc5d-r8z7x\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:20:29.909240 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.909224    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4742f196-ebc8-4851-bad6-ab27dab5f5ac-tls-certs\") pod \"lora-single-adapter-test-kserve-75fdddbc5d-r8z7x\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:20:29.909528 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.909501    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-tmp-dir\") pod \"lora-single-adapter-test-kserve-75fdddbc5d-r8z7x\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:20:29.909591 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.909541    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-home\") pod \"lora-single-adapter-test-kserve-75fdddbc5d-r8z7x\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:20:29.909591 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.909563    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k6s6\" (UniqueName: \"kubernetes.io/projected/4742f196-ebc8-4851-bad6-ab27dab5f5ac-kube-api-access-2k6s6\") pod \"lora-single-adapter-test-kserve-75fdddbc5d-r8z7x\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:20:29.909738 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.909603    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-dshm\") pod \"lora-single-adapter-test-kserve-75fdddbc5d-r8z7x\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:20:29.909738 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.909568    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-kserve-provision-location\") pod \"lora-single-adapter-test-kserve-75fdddbc5d-r8z7x\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:20:29.909738 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.909647    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-model-cache\") pod \"lora-single-adapter-test-kserve-75fdddbc5d-r8z7x\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:20:29.910006 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.909984    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4742f196-ebc8-4851-bad6-ab27dab5f5ac-cabundle-cert\") pod \"lora-single-adapter-test-kserve-75fdddbc5d-r8z7x\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:20:29.911924 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.911900    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-dshm\") pod \"lora-single-adapter-test-kserve-75fdddbc5d-r8z7x\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:20:29.912065 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.912011    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4742f196-ebc8-4851-bad6-ab27dab5f5ac-tls-certs\") pod \"lora-single-adapter-test-kserve-75fdddbc5d-r8z7x\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:20:29.919267 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:29.919241    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k6s6\" (UniqueName: \"kubernetes.io/projected/4742f196-ebc8-4851-bad6-ab27dab5f5ac-kube-api-access-2k6s6\") pod \"lora-single-adapter-test-kserve-75fdddbc5d-r8z7x\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") " pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:20:30.077328 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:30.077224    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:20:30.308993 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:30.308957    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"]
Jul 08 17:20:30.310947 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:20:30.310829    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4742f196_ebc8_4851_bad6_ab27dab5f5ac.slice/crio-773fbe33c5dbd3ddcd1bf69c423155ff461241c68cfa3637d3613eafc7ac79ae WatchSource:0}: Error finding container 773fbe33c5dbd3ddcd1bf69c423155ff461241c68cfa3637d3613eafc7ac79ae: Status 404 returned error can't find the container with id 773fbe33c5dbd3ddcd1bf69c423155ff461241c68cfa3637d3613eafc7ac79ae
Jul 08 17:20:30.694517 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:30.694476    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x" event={"ID":"4742f196-ebc8-4851-bad6-ab27dab5f5ac","Type":"ContainerStarted","Data":"06c56a787826d735cb4b0199a6fa1a7c5863f829c22f302698fc6f661f94dd1f"}
Jul 08 17:20:30.694715 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:30.694540    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x" event={"ID":"4742f196-ebc8-4851-bad6-ab27dab5f5ac","Type":"ContainerStarted","Data":"773fbe33c5dbd3ddcd1bf69c423155ff461241c68cfa3637d3613eafc7ac79ae"}
Jul 08 17:20:34.727021 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:34.726982    2565 generic.go:356] "Generic (PLEG): container finished" podID="49d08e4c-b706-4071-a94c-26aca5a1fa9a" containerID="fe495a985461ffa05a10f7430014a4e7769be9d4927f8b342f311fedfca9c2a1" exitCode=0
Jul 08 17:20:34.727540 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:34.727061    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d" event={"ID":"49d08e4c-b706-4071-a94c-26aca5a1fa9a","Type":"ContainerDied","Data":"fe495a985461ffa05a10f7430014a4e7769be9d4927f8b342f311fedfca9c2a1"}
Jul 08 17:20:34.727540 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:34.727518    2565 scope.go:117] "RemoveContainer" containerID="fe495a985461ffa05a10f7430014a4e7769be9d4927f8b342f311fedfca9c2a1"
Jul 08 17:20:34.728622 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:34.728571    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 08 17:20:34.728709 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:34.728670    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 08 17:20:34.728793 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:34.728745    2565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236220Ki","pods":"250"}
Jul 08 17:20:35.737272 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:35.736357    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d" event={"ID":"49d08e4c-b706-4071-a94c-26aca5a1fa9a","Type":"ContainerStarted","Data":"816599bd008245c7c95aa52a1a33c4df90536a4ea4591e955bf305eae8a541b5"}
Jul 08 17:20:35.781658 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:35.781619    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:35.782927 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:35.782898    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d" podUID="49d08e4c-b706-4071-a94c-26aca5a1fa9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Jul 08 17:20:36.748890 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:36.748828    2565 generic.go:356] "Generic (PLEG): container finished" podID="4742f196-ebc8-4851-bad6-ab27dab5f5ac" containerID="06c56a787826d735cb4b0199a6fa1a7c5863f829c22f302698fc6f661f94dd1f" exitCode=0
Jul 08 17:20:36.750054 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:36.750023    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x" event={"ID":"4742f196-ebc8-4851-bad6-ab27dab5f5ac","Type":"ContainerDied","Data":"06c56a787826d735cb4b0199a6fa1a7c5863f829c22f302698fc6f661f94dd1f"}
Jul 08 17:20:36.781788 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:36.781744    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d" podUID="49d08e4c-b706-4071-a94c-26aca5a1fa9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Jul 08 17:20:37.643301 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:37.643260    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:37.762346 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:37.762300    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x" event={"ID":"4742f196-ebc8-4851-bad6-ab27dab5f5ac","Type":"ContainerStarted","Data":"2987663baf05157b285ed2512d337577a075fd6aebc9ef03c4d0973178f889df"}
Jul 08 17:20:37.781724 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:37.781689    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d" podUID="49d08e4c-b706-4071-a94c-26aca5a1fa9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Jul 08 17:20:37.798130 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:37.798072    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x" podStartSLOduration=3.124446273 podStartE2EDuration="8.798051056s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.673604783s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:20:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:20:37.797141544 +0000 UTC m=+2774.510503007" watchObservedRunningTime="2026-07-08 17:20:37.798051056 +0000 UTC m=+2774.511412485"
Jul 08 17:20:38.078058 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:38.078013    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:20:38.781517 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:38.781475    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d" podUID="49d08e4c-b706-4071-a94c-26aca5a1fa9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Jul 08 17:20:39.443429 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.443381    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"]
Jul 08 17:20:39.447130 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.447106    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:20:39.451520 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.451494    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-self-signed-certs\""
Jul 08 17:20:39.451921 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.451576    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"router-with-refs-pd-test-kserve-dockercfg-lsrbp\""
Jul 08 17:20:39.465342 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.465215    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"]
Jul 08 17:20:39.476667 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.476632    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"]
Jul 08 17:20:39.480162 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.480140    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"
Jul 08 17:20:39.501434 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.501406    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"]
Jul 08 17:20:39.505882 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.505849    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-model-cache\") pod \"router-with-refs-pd-test-kserve-57d4965598-p2stc\" (UID: \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:20:39.506020 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.505947    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-dshm\") pod \"router-with-refs-pd-test-kserve-57d4965598-p2stc\" (UID: \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:20:39.506062 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.506041    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scmr6\" (UniqueName: \"kubernetes.io/projected/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-kube-api-access-scmr6\") pod \"router-with-refs-pd-test-kserve-57d4965598-p2stc\" (UID: \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:20:39.506099 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.506071    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-57d4965598-p2stc\" (UID: \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:20:39.506134 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.506115    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-home\") pod \"router-with-refs-pd-test-kserve-57d4965598-p2stc\" (UID: \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:20:39.506176 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.506160    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-tls-certs\") pod \"router-with-refs-pd-test-kserve-57d4965598-p2stc\" (UID: \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:20:39.506210 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.506193    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-tmp-dir\") pod \"router-with-refs-pd-test-kserve-57d4965598-p2stc\" (UID: \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:20:39.513104 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.513079    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:20:39.607116 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.607084    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-model-cache\") pod \"router-with-refs-pd-test-kserve-57d4965598-p2stc\" (UID: \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:20:39.607333 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.607318    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-dshm\") pod \"router-with-refs-pd-test-kserve-57d4965598-p2stc\" (UID: \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:20:39.607447 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.607437    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scmr6\" (UniqueName: \"kubernetes.io/projected/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-kube-api-access-scmr6\") pod \"router-with-refs-pd-test-kserve-57d4965598-p2stc\" (UID: \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:20:39.607526 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.607501    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-model-cache\") pod \"router-with-refs-pd-test-kserve-57d4965598-p2stc\" (UID: \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:20:39.607728 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.607507    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-57d4965598-p2stc\" (UID: \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:20:39.607876 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.607862    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn\" (UID: \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"
Jul 08 17:20:39.608004 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.607989    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-home\") pod \"router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn\" (UID: \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"
Jul 08 17:20:39.608108 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.608096    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zh56\" (UniqueName: \"kubernetes.io/projected/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-kube-api-access-5zh56\") pod \"router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn\" (UID: \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"
Jul 08 17:20:39.608216 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.608205    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-home\") pod \"router-with-refs-pd-test-kserve-57d4965598-p2stc\" (UID: \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:20:39.608304 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.608294    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn\" (UID: \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"
Jul 08 17:20:39.608422 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.608409    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-tls-certs\") pod \"router-with-refs-pd-test-kserve-57d4965598-p2stc\" (UID: \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:20:39.608516 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.607769    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-57d4965598-p2stc\" (UID: \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:20:39.608583 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.608506    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn\" (UID: \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"
Jul 08 17:20:39.608583 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.608558    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-home\") pod \"router-with-refs-pd-test-kserve-57d4965598-p2stc\" (UID: \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:20:39.608667 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.608578    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn\" (UID: \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"
Jul 08 17:20:39.608667 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.608636    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-tmp-dir\") pod \"router-with-refs-pd-test-kserve-57d4965598-p2stc\" (UID: \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:20:39.608821 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.608702    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn\" (UID: \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"
Jul 08 17:20:39.608934 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.608880    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-tmp-dir\") pod \"router-with-refs-pd-test-kserve-57d4965598-p2stc\" (UID: \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:20:39.609885 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.609863    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-dshm\") pod \"router-with-refs-pd-test-kserve-57d4965598-p2stc\" (UID: \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:20:39.610952 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.610934    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-tls-certs\") pod \"router-with-refs-pd-test-kserve-57d4965598-p2stc\" (UID: \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:20:39.616299 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.616272    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scmr6\" (UniqueName: \"kubernetes.io/projected/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-kube-api-access-scmr6\") pod \"router-with-refs-pd-test-kserve-57d4965598-p2stc\" (UID: \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:20:39.710045 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.709937    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn\" (UID: \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"
Jul 08 17:20:39.710045 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.709986    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-home\") pod \"router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn\" (UID: \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"
Jul 08 17:20:39.710045 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.710014    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zh56\" (UniqueName: \"kubernetes.io/projected/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-kube-api-access-5zh56\") pod \"router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn\" (UID: \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"
Jul 08 17:20:39.710045 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.710044    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn\" (UID: \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"
Jul 08 17:20:39.710439 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.710083    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn\" (UID: \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"
Jul 08 17:20:39.710439 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.710104    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn\" (UID: \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"
Jul 08 17:20:39.710439 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.710142    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn\" (UID: \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"
Jul 08 17:20:39.710657 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.710628    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-tmp-dir\") pod \"router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn\" (UID: \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"
Jul 08 17:20:39.711227 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.711204    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-model-cache\") pod \"router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn\" (UID: \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"
Jul 08 17:20:39.711436 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.711418    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-home\") pod \"router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn\" (UID: \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"
Jul 08 17:20:39.711947 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.711901    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-kserve-provision-location\") pod \"router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn\" (UID: \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"
Jul 08 17:20:39.713262 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.713241    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-tls-certs\") pod \"router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn\" (UID: \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"
Jul 08 17:20:39.713610 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.713592    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-dshm\") pod \"router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn\" (UID: \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"
Jul 08 17:20:39.720962 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.720942    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zh56\" (UniqueName: \"kubernetes.io/projected/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-kube-api-access-5zh56\") pod \"router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn\" (UID: \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\") " pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"
Jul 08 17:20:39.768218 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.768180    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:20:39.782027 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.781983    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d" podUID="49d08e4c-b706-4071-a94c-26aca5a1fa9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Jul 08 17:20:39.797883 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:39.797855    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"
Jul 08 17:20:40.004732 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:40.004703    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"]
Jul 08 17:20:40.006417 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:20:40.006366    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fbec2d8_b74e_4b6c_bf96_1bdcc65ad1ce.slice/crio-b65a25d182e27ab373880d2532525de309ca3cbc9be434b450871511d865d844 WatchSource:0}: Error finding container b65a25d182e27ab373880d2532525de309ca3cbc9be434b450871511d865d844: Status 404 returned error can't find the container with id b65a25d182e27ab373880d2532525de309ca3cbc9be434b450871511d865d844
Jul 08 17:20:40.055422 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:20:40.054877    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod094a3e40_7f9e_4974_8fe5_7ab1fe58f6b1.slice/crio-00364fa83a7c49048dc65b18b001e5f3c15b4aa888fb67075fa8a18d257cd454 WatchSource:0}: Error finding container 00364fa83a7c49048dc65b18b001e5f3c15b4aa888fb67075fa8a18d257cd454: Status 404 returned error can't find the container with id 00364fa83a7c49048dc65b18b001e5f3c15b4aa888fb67075fa8a18d257cd454
Jul 08 17:20:40.085125 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:40.085053    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:20:40.090491 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:40.086251    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x" podUID="4742f196-ebc8-4851-bad6-ab27dab5f5ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Jul 08 17:20:40.096233 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:40.095877    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"]
Jul 08 17:20:40.107279 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:40.107242    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:20:40.173821 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:40.173768    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:20:40.781427 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:40.781383    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d" podUID="49d08e4c-b706-4071-a94c-26aca5a1fa9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Jul 08 17:20:40.786957 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:40.786870    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn" event={"ID":"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1","Type":"ContainerStarted","Data":"287bd8ee438eae93db0f39479d1d276d06ddb9ce03a1f4346f0172be8bf23ac0"}
Jul 08 17:20:40.786957 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:40.786929    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn" event={"ID":"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1","Type":"ContainerStarted","Data":"00364fa83a7c49048dc65b18b001e5f3c15b4aa888fb67075fa8a18d257cd454"}
Jul 08 17:20:40.788927 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:40.788894    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" event={"ID":"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce","Type":"ContainerStarted","Data":"6f2625b0ead782216faeb2902899183902bea722102ac55c4d96088771a7e989"}
Jul 08 17:20:40.789054 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:40.788951    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" event={"ID":"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce","Type":"ContainerStarted","Data":"b65a25d182e27ab373880d2532525de309ca3cbc9be434b450871511d865d844"}
Jul 08 17:20:40.789054 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:40.789036    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:20:41.786809 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:41.786771    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:41.788778 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:41.788748    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-gateway-2-openshift-default-684dc9b4b9-h2x2d"
Jul 08 17:20:41.807905 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:41.807661    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" event={"ID":"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce","Type":"ContainerStarted","Data":"57b7a778deaf9f02f524088f4aebc3a9471ab15ed14f016f518135b0ff46bc61"}
Jul 08 17:20:44.831696 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:44.831571    2565 generic.go:356] "Generic (PLEG): container finished" podID="094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" containerID="287bd8ee438eae93db0f39479d1d276d06ddb9ce03a1f4346f0172be8bf23ac0" exitCode=0
Jul 08 17:20:44.832126 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:44.831732    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn" event={"ID":"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1","Type":"ContainerDied","Data":"287bd8ee438eae93db0f39479d1d276d06ddb9ce03a1f4346f0172be8bf23ac0"}
Jul 08 17:20:45.840763 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:45.840635    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn" event={"ID":"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1","Type":"ContainerStarted","Data":"9133d92c96d34c22e087e76375771f28d5cffa66f28516627fc63083596fe9b2"}
Jul 08 17:20:45.842812 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:45.842784    2565 generic.go:356] "Generic (PLEG): container finished" podID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerID="57b7a778deaf9f02f524088f4aebc3a9471ab15ed14f016f518135b0ff46bc61" exitCode=0
Jul 08 17:20:45.842972 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:45.842829    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" event={"ID":"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce","Type":"ContainerDied","Data":"57b7a778deaf9f02f524088f4aebc3a9471ab15ed14f016f518135b0ff46bc61"}
Jul 08 17:20:45.871843 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:45.871785    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn" podStartSLOduration=2.532312568 podStartE2EDuration="6.871763297s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.339450729s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:20:45.869049148 +0000 UTC m=+2782.582410590" watchObservedRunningTime="2026-07-08 17:20:45.871763297 +0000 UTC m=+2782.585124726"
Jul 08 17:20:46.857194 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:46.857144    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" event={"ID":"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce","Type":"ContainerStarted","Data":"2b71925dd07595029f27842341ccf4af12bd7181d18315e55885921f5da44011"}
Jul 08 17:20:46.885996 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:46.885929    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" podStartSLOduration=3.251901842 podStartE2EDuration="7.885904767s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.634002925s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:20:46.884216901 +0000 UTC m=+2783.597578364" watchObservedRunningTime="2026-07-08 17:20:46.885904767 +0000 UTC m=+2783.599266195"
Jul 08 17:20:49.769300 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:49.769249    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:20:49.769785 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:49.769346    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:20:49.770773 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:49.770746    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Jul 08 17:20:49.799100 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:49.799064    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"
Jul 08 17:20:49.799230 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:49.799140    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"
Jul 08 17:20:49.800839 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:49.800811    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn" podUID="094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 08 17:20:50.078600 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:50.078485    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x" podUID="4742f196-ebc8-4851-bad6-ab27dab5f5ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Jul 08 17:20:59.769326 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:59.769276    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Jul 08 17:20:59.788101 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:59.788070    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:20:59.799066 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:20:59.799021    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn" podUID="094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 08 17:21:00.078745 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:21:00.078618    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x" podUID="4742f196-ebc8-4851-bad6-ab27dab5f5ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Jul 08 17:21:09.769127 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:21:09.769078    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Jul 08 17:21:09.798901 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:21:09.798847    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn" podUID="094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 08 17:21:10.078841 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:21:10.078740    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x" podUID="4742f196-ebc8-4851-bad6-ab27dab5f5ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Jul 08 17:21:19.769311 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:21:19.769264    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Jul 08 17:21:19.798763 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:21:19.798711    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn" podUID="094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 08 17:21:20.078457 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:21:20.078326    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x" podUID="4742f196-ebc8-4851-bad6-ab27dab5f5ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Jul 08 17:21:23.784067 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:21:23.784023    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:21:29.769503 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:21:29.769451    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Jul 08 17:21:29.798881 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:21:29.798836    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn" podUID="094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 08 17:21:30.077808 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:21:30.077690    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x" podUID="4742f196-ebc8-4851-bad6-ab27dab5f5ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Jul 08 17:21:39.769576 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:21:39.769528    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Jul 08 17:21:39.798859 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:21:39.798802    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn" podUID="094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 08 17:21:40.078292 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:21:40.078180    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x" podUID="4742f196-ebc8-4851-bad6-ab27dab5f5ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Jul 08 17:21:49.769559 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:21:49.769497    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Jul 08 17:21:49.799101 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:21:49.799048    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn" podUID="094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 08 17:21:50.078208 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:21:50.078081    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x" podUID="4742f196-ebc8-4851-bad6-ab27dab5f5ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Jul 08 17:21:59.769056 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:21:59.769003    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Jul 08 17:21:59.799184 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:21:59.799149    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn" podUID="094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 08 17:22:00.091309 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:22:00.091217    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:22:00.100450 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:22:00.100416    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:22:09.769433 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:22:09.769385    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Jul 08 17:22:09.799197 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:22:09.799158    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn" podUID="094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 08 17:22:15.509419 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:22:15.509309    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:22:15.610566 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:22:15.610530    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:22:19.768984 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:22:19.768939    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Jul 08 17:22:19.799099 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:22:19.799060    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn" podUID="094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 08 17:22:29.769857 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:22:29.769802    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Jul 08 17:22:29.799254 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:22:29.799207    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn" podUID="094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 08 17:22:39.769537 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:22:39.769489    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Jul 08 17:22:39.798855 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:22:39.798815    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn" podUID="094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 08 17:22:49.769505 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:22:49.769438    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Jul 08 17:22:49.799212 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:22:49.799166    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn" podUID="094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 08 17:22:59.769588 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:22:59.769549    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Jul 08 17:22:59.798414 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:22:59.798367    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn" podUID="094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 08 17:23:09.769432 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:09.769381    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Jul 08 17:23:09.798463 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:09.798428    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn" podUID="094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 08 17:23:19.768921 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:19.768872    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Jul 08 17:23:19.799011 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:19.798964    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn" podUID="094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 08 17:23:27.922397 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:27.922351    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"]
Jul 08 17:23:27.958014 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:27.957980    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:23:29.769265 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:29.769220    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Jul 08 17:23:29.798987 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:29.798946    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn" podUID="094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 08 17:23:39.769596 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:39.769535    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Jul 08 17:23:39.799046 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:39.798989    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn" podUID="094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": dial tcp 10.133.0.74:8000: connect: connection refused"
Jul 08 17:23:42.947871 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:42.947828    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x" podUID="4742f196-ebc8-4851-bad6-ab27dab5f5ac" containerName="main" containerID="cri-o://2987663baf05157b285ed2512d337577a075fd6aebc9ef03c4d0973178f889df" gracePeriod=45
Jul 08 17:23:43.092389 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:43.092341    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x" podUID="4742f196-ebc8-4851-bad6-ab27dab5f5ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": dial tcp 10.133.0.72:8000: connect: connection refused"
Jul 08 17:23:44.010865 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:44.010825    2565 generic.go:356] "Generic (PLEG): container finished" podID="4742f196-ebc8-4851-bad6-ab27dab5f5ac" containerID="2987663baf05157b285ed2512d337577a075fd6aebc9ef03c4d0973178f889df" exitCode=0
Jul 08 17:23:44.011204 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:44.010886    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x" event={"ID":"4742f196-ebc8-4851-bad6-ab27dab5f5ac","Type":"ContainerDied","Data":"2987663baf05157b285ed2512d337577a075fd6aebc9ef03c4d0973178f889df"}
Jul 08 17:23:44.119822 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:44.119768    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:23:44.154563 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:44.154541    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-dshm\") pod \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") "
Jul 08 17:23:44.154738 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:44.154585    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4742f196-ebc8-4851-bad6-ab27dab5f5ac-cabundle-cert\") pod \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") "
Jul 08 17:23:44.154738 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:44.154603    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-kserve-provision-location\") pod \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") "
Jul 08 17:23:44.154738 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:44.154619    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-model-cache\") pod \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") "
Jul 08 17:23:44.154738 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:44.154645    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4742f196-ebc8-4851-bad6-ab27dab5f5ac-tls-certs\") pod \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") "
Jul 08 17:23:44.154986 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:44.154749    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2k6s6\" (UniqueName: \"kubernetes.io/projected/4742f196-ebc8-4851-bad6-ab27dab5f5ac-kube-api-access-2k6s6\") pod \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") "
Jul 08 17:23:44.154986 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:44.154799    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-tmp-dir\") pod \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") "
Jul 08 17:23:44.154986 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:44.154826    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-home\") pod \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\" (UID: \"4742f196-ebc8-4851-bad6-ab27dab5f5ac\") "
Jul 08 17:23:44.154986 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:44.154855    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "4742f196-ebc8-4851-bad6-ab27dab5f5ac" (UID: "4742f196-ebc8-4851-bad6-ab27dab5f5ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:23:44.154986 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:44.154959    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4742f196-ebc8-4851-bad6-ab27dab5f5ac-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4742f196-ebc8-4851-bad6-ab27dab5f5ac" (UID: "4742f196-ebc8-4851-bad6-ab27dab5f5ac"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 17:23:44.155267 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:44.155184    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-home" (OuterVolumeSpecName: "home") pod "4742f196-ebc8-4851-bad6-ab27dab5f5ac" (UID: "4742f196-ebc8-4851-bad6-ab27dab5f5ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:23:44.155337 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:44.155274    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4742f196-ebc8-4851-bad6-ab27dab5f5ac-cabundle-cert\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:23:44.155337 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:44.155294    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:23:44.155337 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:44.155307    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:23:44.161880 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:44.161857    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-dshm" (OuterVolumeSpecName: "dshm") pod "4742f196-ebc8-4851-bad6-ab27dab5f5ac" (UID: "4742f196-ebc8-4851-bad6-ab27dab5f5ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:23:44.161979 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:44.161883    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4742f196-ebc8-4851-bad6-ab27dab5f5ac-kube-api-access-2k6s6" (OuterVolumeSpecName: "kube-api-access-2k6s6") pod "4742f196-ebc8-4851-bad6-ab27dab5f5ac" (UID: "4742f196-ebc8-4851-bad6-ab27dab5f5ac"). InnerVolumeSpecName "kube-api-access-2k6s6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:23:44.161979 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:44.161953    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4742f196-ebc8-4851-bad6-ab27dab5f5ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4742f196-ebc8-4851-bad6-ab27dab5f5ac" (UID: "4742f196-ebc8-4851-bad6-ab27dab5f5ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:23:44.171649 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:44.171613    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4742f196-ebc8-4851-bad6-ab27dab5f5ac" (UID: "4742f196-ebc8-4851-bad6-ab27dab5f5ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:23:44.225743 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:44.225712    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4742f196-ebc8-4851-bad6-ab27dab5f5ac" (UID: "4742f196-ebc8-4851-bad6-ab27dab5f5ac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:23:44.255894 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:44.255867    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:23:44.255894 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:44.255892    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-kserve-provision-location\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:23:44.256039 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:44.255907    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4742f196-ebc8-4851-bad6-ab27dab5f5ac-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:23:44.256039 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:44.255919    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k6s6\" (UniqueName: \"kubernetes.io/projected/4742f196-ebc8-4851-bad6-ab27dab5f5ac-kube-api-access-2k6s6\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:23:44.256039 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:44.255933    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4742f196-ebc8-4851-bad6-ab27dab5f5ac-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:23:45.018264 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:45.018189    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"
Jul 08 17:23:45.018264 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:45.018213    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x" event={"ID":"4742f196-ebc8-4851-bad6-ab27dab5f5ac","Type":"ContainerDied","Data":"773fbe33c5dbd3ddcd1bf69c423155ff461241c68cfa3637d3613eafc7ac79ae"}
Jul 08 17:23:45.018870 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:45.018276    2565 scope.go:117] "RemoveContainer" containerID="2987663baf05157b285ed2512d337577a075fd6aebc9ef03c4d0973178f889df"
Jul 08 17:23:45.028277 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:45.028259    2565 scope.go:117] "RemoveContainer" containerID="06c56a787826d735cb4b0199a6fa1a7c5863f829c22f302698fc6f661f94dd1f"
Jul 08 17:23:45.043775 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:45.043750    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"]
Jul 08 17:23:45.051178 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:45.051151    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x"]
Jul 08 17:23:45.092425 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:45.092386    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/lora-single-adapter-test-kserve-75fdddbc5d-r8z7x" podUID="4742f196-ebc8-4851-bad6-ab27dab5f5ac" containerName="main" probeResult="failure" output="Get \"https://10.133.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 08 17:23:45.822838 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:45.822804    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4742f196-ebc8-4851-bad6-ab27dab5f5ac" path="/var/lib/kubelet/pods/4742f196-ebc8-4851-bad6-ab27dab5f5ac/volumes"
Jul 08 17:23:49.768817 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:49.768774    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerName="main" probeResult="failure" output="Get \"https://10.133.0.73:8001/health\": dial tcp 10.133.0.73:8001: connect: connection refused"
Jul 08 17:23:49.808917 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:49.808889    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"
Jul 08 17:23:49.821995 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:49.821970    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"
Jul 08 17:23:59.778945 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:59.778903    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:23:59.790808 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:23:59.790780    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:24:01.677936 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.677901    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"]
Jul 08 17:24:01.678327 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.678304    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4742f196-ebc8-4851-bad6-ab27dab5f5ac" containerName="storage-initializer"
Jul 08 17:24:01.678327 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.678316    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4742f196-ebc8-4851-bad6-ab27dab5f5ac" containerName="storage-initializer"
Jul 08 17:24:01.678327 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.678327    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4742f196-ebc8-4851-bad6-ab27dab5f5ac" containerName="main"
Jul 08 17:24:01.678481 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.678333    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="4742f196-ebc8-4851-bad6-ab27dab5f5ac" containerName="main"
Jul 08 17:24:01.678481 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.678437    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="4742f196-ebc8-4851-bad6-ab27dab5f5ac" containerName="main"
Jul 08 17:24:01.683096 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.683076    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:24:01.685662 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.685642    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 08 17:24:01.685915 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.685899    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"lora-multiple-adapters-test-kserve-self-signed-certs\""
Jul 08 17:24:01.695576 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.695551    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"]
Jul 08 17:24:01.712170 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.712143    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:24:01.804185 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.804149    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33d141fe-7fff-4b18-8e2c-1d9f61bed747-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-7c8db66c87-pzn59\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:24:01.804350 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.804208    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-model-cache\") pod \"lora-multiple-adapters-test-kserve-7c8db66c87-pzn59\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:24:01.804350 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.804271    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-home\") pod \"lora-multiple-adapters-test-kserve-7c8db66c87-pzn59\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:24:01.804350 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.804293    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-dshm\") pod \"lora-multiple-adapters-test-kserve-7c8db66c87-pzn59\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:24:01.804350 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.804317    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-7c8db66c87-pzn59\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:24:01.804350 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.804341    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkcm8\" (UniqueName: \"kubernetes.io/projected/33d141fe-7fff-4b18-8e2c-1d9f61bed747-kube-api-access-gkcm8\") pod \"lora-multiple-adapters-test-kserve-7c8db66c87-pzn59\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:24:01.804563 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.804398    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33d141fe-7fff-4b18-8e2c-1d9f61bed747-tls-certs\") pod \"lora-multiple-adapters-test-kserve-7c8db66c87-pzn59\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:24:01.804563 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.804432    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-7c8db66c87-pzn59\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:24:01.905403 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.905367    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33d141fe-7fff-4b18-8e2c-1d9f61bed747-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-7c8db66c87-pzn59\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:24:01.905566 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.905432    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-model-cache\") pod \"lora-multiple-adapters-test-kserve-7c8db66c87-pzn59\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:24:01.905566 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.905477    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-home\") pod \"lora-multiple-adapters-test-kserve-7c8db66c87-pzn59\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:24:01.905566 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.905499    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-dshm\") pod \"lora-multiple-adapters-test-kserve-7c8db66c87-pzn59\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:24:01.905566 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.905526    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-7c8db66c87-pzn59\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:24:01.905566 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.905555    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkcm8\" (UniqueName: \"kubernetes.io/projected/33d141fe-7fff-4b18-8e2c-1d9f61bed747-kube-api-access-gkcm8\") pod \"lora-multiple-adapters-test-kserve-7c8db66c87-pzn59\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:24:01.905869 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.905592    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33d141fe-7fff-4b18-8e2c-1d9f61bed747-tls-certs\") pod \"lora-multiple-adapters-test-kserve-7c8db66c87-pzn59\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:24:01.905869 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.905616    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-7c8db66c87-pzn59\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:24:01.906322 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.906035    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-kserve-provision-location\") pod \"lora-multiple-adapters-test-kserve-7c8db66c87-pzn59\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:24:01.906322 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.906117    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-tmp-dir\") pod \"lora-multiple-adapters-test-kserve-7c8db66c87-pzn59\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:24:01.906322 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.906150    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-model-cache\") pod \"lora-multiple-adapters-test-kserve-7c8db66c87-pzn59\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:24:01.906322 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.906183    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33d141fe-7fff-4b18-8e2c-1d9f61bed747-cabundle-cert\") pod \"lora-multiple-adapters-test-kserve-7c8db66c87-pzn59\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:24:01.906322 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.906189    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-home\") pod \"lora-multiple-adapters-test-kserve-7c8db66c87-pzn59\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:24:01.908322 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.908279    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-dshm\") pod \"lora-multiple-adapters-test-kserve-7c8db66c87-pzn59\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:24:01.908566 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.908545    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33d141fe-7fff-4b18-8e2c-1d9f61bed747-tls-certs\") pod \"lora-multiple-adapters-test-kserve-7c8db66c87-pzn59\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:24:01.915757 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.915736    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkcm8\" (UniqueName: \"kubernetes.io/projected/33d141fe-7fff-4b18-8e2c-1d9f61bed747-kube-api-access-gkcm8\") pod \"lora-multiple-adapters-test-kserve-7c8db66c87-pzn59\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") " pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:24:01.994517 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:01.994452    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:24:02.144227 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:02.144206    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"]
Jul 08 17:24:02.147181 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:24:02.147137    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33d141fe_7fff_4b18_8e2c_1d9f61bed747.slice/crio-e9b8d796c4be66bf50112e93213e2d279a485a1a0d2e5bf63af7aa9ed9927ab4 WatchSource:0}: Error finding container e9b8d796c4be66bf50112e93213e2d279a485a1a0d2e5bf63af7aa9ed9927ab4: Status 404 returned error can't find the container with id e9b8d796c4be66bf50112e93213e2d279a485a1a0d2e5bf63af7aa9ed9927ab4
Jul 08 17:24:03.125200 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:03.125152    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59" event={"ID":"33d141fe-7fff-4b18-8e2c-1d9f61bed747","Type":"ContainerStarted","Data":"cede1513c382a80fd4a97e92cddb870468c9877b0942dd34e6e030b4d2387b2c"}
Jul 08 17:24:03.125598 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:03.125219    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59" event={"ID":"33d141fe-7fff-4b18-8e2c-1d9f61bed747","Type":"ContainerStarted","Data":"e9b8d796c4be66bf50112e93213e2d279a485a1a0d2e5bf63af7aa9ed9927ab4"}
Jul 08 17:24:07.152836 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:07.152808    2565 generic.go:356] "Generic (PLEG): container finished" podID="33d141fe-7fff-4b18-8e2c-1d9f61bed747" containerID="cede1513c382a80fd4a97e92cddb870468c9877b0942dd34e6e030b4d2387b2c" exitCode=0
Jul 08 17:24:07.153111 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:07.152881    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59" event={"ID":"33d141fe-7fff-4b18-8e2c-1d9f61bed747","Type":"ContainerDied","Data":"cede1513c382a80fd4a97e92cddb870468c9877b0942dd34e6e030b4d2387b2c"}
Jul 08 17:24:08.161969 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:08.161917    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59" event={"ID":"33d141fe-7fff-4b18-8e2c-1d9f61bed747","Type":"ContainerStarted","Data":"5590eed9b2605d663f3c66d21e282611053b4e6688700c71a432b12226d5e6e7"}
Jul 08 17:24:08.188788 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:08.188730    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59" podStartSLOduration=2.269403375 podStartE2EDuration="7.188715478s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.919312103s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:24:08.186266879 +0000 UTC m=+2984.899628328" watchObservedRunningTime="2026-07-08 17:24:08.188715478 +0000 UTC m=+2984.902076903"
Jul 08 17:24:08.995386 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:08.995347    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:24:11.994850 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:11.994814    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:24:11.996737 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:11.996705    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59" podUID="33d141fe-7fff-4b18-8e2c-1d9f61bed747" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Jul 08 17:24:13.632488 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:13.632184    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"]
Jul 08 17:24:13.641274 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:13.641242    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"]
Jul 08 17:24:13.643978 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:13.643950    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:24:13.712963 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:13.712926    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:24:21.995630 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:21.995584    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59" podUID="33d141fe-7fff-4b18-8e2c-1d9f61bed747" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Jul 08 17:24:28.663043 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:28.662969    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn" podUID="094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" containerName="main" containerID="cri-o://9133d92c96d34c22e087e76375771f28d5cffa66f28516627fc63083596fe9b2" gracePeriod=45
Jul 08 17:24:28.670402 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:28.670339    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerName="main" containerID="cri-o://2b71925dd07595029f27842341ccf4af12bd7181d18315e55885921f5da44011" gracePeriod=45
Jul 08 17:24:29.481483 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.481420    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerName="llm-d-routing-sidecar" containerID="cri-o://6f2625b0ead782216faeb2902899183902bea722102ac55c4d96088771a7e989" gracePeriod=45
Jul 08 17:24:29.631405 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.631380    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"
Jul 08 17:24:29.675761 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.675732    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-dshm\") pod \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\" (UID: \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\") "
Jul 08 17:24:29.676175 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.675812    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zh56\" (UniqueName: \"kubernetes.io/projected/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-kube-api-access-5zh56\") pod \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\" (UID: \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\") "
Jul 08 17:24:29.676175 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.675895    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-tmp-dir\") pod \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\" (UID: \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\") "
Jul 08 17:24:29.676175 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.675923    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-tls-certs\") pod \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\" (UID: \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\") "
Jul 08 17:24:29.676175 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.675984    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-home\") pod \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\" (UID: \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\") "
Jul 08 17:24:29.676175 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.676014    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-kserve-provision-location\") pod \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\" (UID: \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\") "
Jul 08 17:24:29.676175 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.676082    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-model-cache\") pod \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\" (UID: \"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1\") "
Jul 08 17:24:29.676602 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.676572    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-model-cache" (OuterVolumeSpecName: "model-cache") pod "094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" (UID: "094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:24:29.676698 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.676635    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-home" (OuterVolumeSpecName: "home") pod "094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" (UID: "094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:24:29.678706 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.678655    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-dshm" (OuterVolumeSpecName: "dshm") pod "094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" (UID: "094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:24:29.678905 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.678876    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" (UID: "094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:24:29.679009 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.678992    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-kube-api-access-5zh56" (OuterVolumeSpecName: "kube-api-access-5zh56") pod "094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" (UID: "094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1"). InnerVolumeSpecName "kube-api-access-5zh56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:24:29.695052 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.695025    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" (UID: "094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:24:29.751211 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.751184    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:24:29.752332 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.752302    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" (UID: "094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:24:29.776808 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.776786    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-scmr6\" (UniqueName: \"kubernetes.io/projected/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-kube-api-access-scmr6\") pod \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\" (UID: \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\") "
Jul 08 17:24:29.776919 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.776821    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-kserve-provision-location\") pod \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\" (UID: \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\") "
Jul 08 17:24:29.776919 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.776885    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-model-cache\") pod \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\" (UID: \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\") "
Jul 08 17:24:29.777041 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.776935    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-tls-certs\") pod \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\" (UID: \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\") "
Jul 08 17:24:29.777041 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.776973    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-dshm\") pod \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\" (UID: \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\") "
Jul 08 17:24:29.777041 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.777016    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-tmp-dir\") pod \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\" (UID: \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\") "
Jul 08 17:24:29.777196 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.777059    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-home\") pod \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\" (UID: \"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce\") "
Jul 08 17:24:29.777539 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.777314    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-model-cache" (OuterVolumeSpecName: "model-cache") pod "2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" (UID: "2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:24:29.777539 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.777415    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zh56\" (UniqueName: \"kubernetes.io/projected/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-kube-api-access-5zh56\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:24:29.777539 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.777435    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:24:29.777539 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.777450    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:24:29.777539 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.777462    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:24:29.777539 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.777475    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-kserve-provision-location\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:24:29.777539 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.777486    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:24:29.777539 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.777498    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:24:29.777539 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.777509    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:24:29.778056 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.777887    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-home" (OuterVolumeSpecName: "home") pod "2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" (UID: "2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:24:29.780347 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.780241    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-kube-api-access-scmr6" (OuterVolumeSpecName: "kube-api-access-scmr6") pod "2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" (UID: "2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce"). InnerVolumeSpecName "kube-api-access-scmr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:24:29.780347 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.780269    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-dshm" (OuterVolumeSpecName: "dshm") pod "2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" (UID: "2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:24:29.780768 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.780663    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" (UID: "2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:24:29.797524 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.797491    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" (UID: "2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:24:29.846368 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.846330    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" (UID: "2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:24:29.878366 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.878326    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:24:29.878366 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.878357    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:24:29.878366 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.878370    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scmr6\" (UniqueName: \"kubernetes.io/projected/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-kube-api-access-scmr6\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:24:29.878725 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.878392    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-kserve-provision-location\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:24:29.878725 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.878405    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:24:29.878725 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:29.878435    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:24:30.299765 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.299731    2565 generic.go:356] "Generic (PLEG): container finished" podID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerID="2b71925dd07595029f27842341ccf4af12bd7181d18315e55885921f5da44011" exitCode=0
Jul 08 17:24:30.299765 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.299758    2565 generic.go:356] "Generic (PLEG): container finished" podID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerID="6f2625b0ead782216faeb2902899183902bea722102ac55c4d96088771a7e989" exitCode=0
Jul 08 17:24:30.300030 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.299812    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"
Jul 08 17:24:30.300030 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.299812    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" event={"ID":"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce","Type":"ContainerDied","Data":"2b71925dd07595029f27842341ccf4af12bd7181d18315e55885921f5da44011"}
Jul 08 17:24:30.300030 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.299907    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" event={"ID":"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce","Type":"ContainerDied","Data":"6f2625b0ead782216faeb2902899183902bea722102ac55c4d96088771a7e989"}
Jul 08 17:24:30.300030 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.299933    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc" event={"ID":"2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce","Type":"ContainerDied","Data":"b65a25d182e27ab373880d2532525de309ca3cbc9be434b450871511d865d844"}
Jul 08 17:24:30.300030 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.299965    2565 scope.go:117] "RemoveContainer" containerID="2b71925dd07595029f27842341ccf4af12bd7181d18315e55885921f5da44011"
Jul 08 17:24:30.301699 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.301656    2565 generic.go:356] "Generic (PLEG): container finished" podID="094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" containerID="9133d92c96d34c22e087e76375771f28d5cffa66f28516627fc63083596fe9b2" exitCode=0
Jul 08 17:24:30.301812 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.301712    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn" event={"ID":"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1","Type":"ContainerDied","Data":"9133d92c96d34c22e087e76375771f28d5cffa66f28516627fc63083596fe9b2"}
Jul 08 17:24:30.301812 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.301746    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"
Jul 08 17:24:30.301812 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.301757    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn" event={"ID":"094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1","Type":"ContainerDied","Data":"00364fa83a7c49048dc65b18b001e5f3c15b4aa888fb67075fa8a18d257cd454"}
Jul 08 17:24:30.310272 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.310252    2565 scope.go:117] "RemoveContainer" containerID="57b7a778deaf9f02f524088f4aebc3a9471ab15ed14f016f518135b0ff46bc61"
Jul 08 17:24:30.318253 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.318233    2565 scope.go:117] "RemoveContainer" containerID="6f2625b0ead782216faeb2902899183902bea722102ac55c4d96088771a7e989"
Jul 08 17:24:30.324997 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.324976    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"]
Jul 08 17:24:30.326530 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.326515    2565 scope.go:117] "RemoveContainer" containerID="2b71925dd07595029f27842341ccf4af12bd7181d18315e55885921f5da44011"
Jul 08 17:24:30.326808 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:24:30.326779    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b71925dd07595029f27842341ccf4af12bd7181d18315e55885921f5da44011\": container with ID starting with 2b71925dd07595029f27842341ccf4af12bd7181d18315e55885921f5da44011 not found: ID does not exist" containerID="2b71925dd07595029f27842341ccf4af12bd7181d18315e55885921f5da44011"
Jul 08 17:24:30.326937 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.326847    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b71925dd07595029f27842341ccf4af12bd7181d18315e55885921f5da44011"} err="failed to get container status \"2b71925dd07595029f27842341ccf4af12bd7181d18315e55885921f5da44011\": rpc error: code = NotFound desc = could not find container \"2b71925dd07595029f27842341ccf4af12bd7181d18315e55885921f5da44011\": container with ID starting with 2b71925dd07595029f27842341ccf4af12bd7181d18315e55885921f5da44011 not found: ID does not exist"
Jul 08 17:24:30.326937 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.326871    2565 scope.go:117] "RemoveContainer" containerID="57b7a778deaf9f02f524088f4aebc3a9471ab15ed14f016f518135b0ff46bc61"
Jul 08 17:24:30.327143 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:24:30.327118    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57b7a778deaf9f02f524088f4aebc3a9471ab15ed14f016f518135b0ff46bc61\": container with ID starting with 57b7a778deaf9f02f524088f4aebc3a9471ab15ed14f016f518135b0ff46bc61 not found: ID does not exist" containerID="57b7a778deaf9f02f524088f4aebc3a9471ab15ed14f016f518135b0ff46bc61"
Jul 08 17:24:30.327221 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.327151    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b7a778deaf9f02f524088f4aebc3a9471ab15ed14f016f518135b0ff46bc61"} err="failed to get container status \"57b7a778deaf9f02f524088f4aebc3a9471ab15ed14f016f518135b0ff46bc61\": rpc error: code = NotFound desc = could not find container \"57b7a778deaf9f02f524088f4aebc3a9471ab15ed14f016f518135b0ff46bc61\": container with ID starting with 57b7a778deaf9f02f524088f4aebc3a9471ab15ed14f016f518135b0ff46bc61 not found: ID does not exist"
Jul 08 17:24:30.327221 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.327174    2565 scope.go:117] "RemoveContainer" containerID="6f2625b0ead782216faeb2902899183902bea722102ac55c4d96088771a7e989"
Jul 08 17:24:30.327789 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:24:30.327593    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f2625b0ead782216faeb2902899183902bea722102ac55c4d96088771a7e989\": container with ID starting with 6f2625b0ead782216faeb2902899183902bea722102ac55c4d96088771a7e989 not found: ID does not exist" containerID="6f2625b0ead782216faeb2902899183902bea722102ac55c4d96088771a7e989"
Jul 08 17:24:30.327789 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.327626    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2625b0ead782216faeb2902899183902bea722102ac55c4d96088771a7e989"} err="failed to get container status \"6f2625b0ead782216faeb2902899183902bea722102ac55c4d96088771a7e989\": rpc error: code = NotFound desc = could not find container \"6f2625b0ead782216faeb2902899183902bea722102ac55c4d96088771a7e989\": container with ID starting with 6f2625b0ead782216faeb2902899183902bea722102ac55c4d96088771a7e989 not found: ID does not exist"
Jul 08 17:24:30.327789 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.327647    2565 scope.go:117] "RemoveContainer" containerID="2b71925dd07595029f27842341ccf4af12bd7181d18315e55885921f5da44011"
Jul 08 17:24:30.328035 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.327985    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b71925dd07595029f27842341ccf4af12bd7181d18315e55885921f5da44011"} err="failed to get container status \"2b71925dd07595029f27842341ccf4af12bd7181d18315e55885921f5da44011\": rpc error: code = NotFound desc = could not find container \"2b71925dd07595029f27842341ccf4af12bd7181d18315e55885921f5da44011\": container with ID starting with 2b71925dd07595029f27842341ccf4af12bd7181d18315e55885921f5da44011 not found: ID does not exist"
Jul 08 17:24:30.328035 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.328005    2565 scope.go:117] "RemoveContainer" containerID="57b7a778deaf9f02f524088f4aebc3a9471ab15ed14f016f518135b0ff46bc61"
Jul 08 17:24:30.328311 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.328281    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b7a778deaf9f02f524088f4aebc3a9471ab15ed14f016f518135b0ff46bc61"} err="failed to get container status \"57b7a778deaf9f02f524088f4aebc3a9471ab15ed14f016f518135b0ff46bc61\": rpc error: code = NotFound desc = could not find container \"57b7a778deaf9f02f524088f4aebc3a9471ab15ed14f016f518135b0ff46bc61\": container with ID starting with 57b7a778deaf9f02f524088f4aebc3a9471ab15ed14f016f518135b0ff46bc61 not found: ID does not exist"
Jul 08 17:24:30.328311 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.328304    2565 scope.go:117] "RemoveContainer" containerID="6f2625b0ead782216faeb2902899183902bea722102ac55c4d96088771a7e989"
Jul 08 17:24:30.328576 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.328551    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2625b0ead782216faeb2902899183902bea722102ac55c4d96088771a7e989"} err="failed to get container status \"6f2625b0ead782216faeb2902899183902bea722102ac55c4d96088771a7e989\": rpc error: code = NotFound desc = could not find container \"6f2625b0ead782216faeb2902899183902bea722102ac55c4d96088771a7e989\": container with ID starting with 6f2625b0ead782216faeb2902899183902bea722102ac55c4d96088771a7e989 not found: ID does not exist"
Jul 08 17:24:30.328576 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.328576    2565 scope.go:117] "RemoveContainer" containerID="9133d92c96d34c22e087e76375771f28d5cffa66f28516627fc63083596fe9b2"
Jul 08 17:24:30.330487 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.330467    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-prefill-5c9fc7cc79-2ggqn"]
Jul 08 17:24:30.336301 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.336284    2565 scope.go:117] "RemoveContainer" containerID="287bd8ee438eae93db0f39479d1d276d06ddb9ce03a1f4346f0172be8bf23ac0"
Jul 08 17:24:30.344367 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.344350    2565 scope.go:117] "RemoveContainer" containerID="9133d92c96d34c22e087e76375771f28d5cffa66f28516627fc63083596fe9b2"
Jul 08 17:24:30.344612 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:24:30.344595    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9133d92c96d34c22e087e76375771f28d5cffa66f28516627fc63083596fe9b2\": container with ID starting with 9133d92c96d34c22e087e76375771f28d5cffa66f28516627fc63083596fe9b2 not found: ID does not exist" containerID="9133d92c96d34c22e087e76375771f28d5cffa66f28516627fc63083596fe9b2"
Jul 08 17:24:30.344663 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.344617    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9133d92c96d34c22e087e76375771f28d5cffa66f28516627fc63083596fe9b2"} err="failed to get container status \"9133d92c96d34c22e087e76375771f28d5cffa66f28516627fc63083596fe9b2\": rpc error: code = NotFound desc = could not find container \"9133d92c96d34c22e087e76375771f28d5cffa66f28516627fc63083596fe9b2\": container with ID starting with 9133d92c96d34c22e087e76375771f28d5cffa66f28516627fc63083596fe9b2 not found: ID does not exist"
Jul 08 17:24:30.344663 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.344631    2565 scope.go:117] "RemoveContainer" containerID="287bd8ee438eae93db0f39479d1d276d06ddb9ce03a1f4346f0172be8bf23ac0"
Jul 08 17:24:30.344879 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:24:30.344860    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"287bd8ee438eae93db0f39479d1d276d06ddb9ce03a1f4346f0172be8bf23ac0\": container with ID starting with 287bd8ee438eae93db0f39479d1d276d06ddb9ce03a1f4346f0172be8bf23ac0 not found: ID does not exist" containerID="287bd8ee438eae93db0f39479d1d276d06ddb9ce03a1f4346f0172be8bf23ac0"
Jul 08 17:24:30.344926 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.344882    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287bd8ee438eae93db0f39479d1d276d06ddb9ce03a1f4346f0172be8bf23ac0"} err="failed to get container status \"287bd8ee438eae93db0f39479d1d276d06ddb9ce03a1f4346f0172be8bf23ac0\": rpc error: code = NotFound desc = could not find container \"287bd8ee438eae93db0f39479d1d276d06ddb9ce03a1f4346f0172be8bf23ac0\": container with ID starting with 287bd8ee438eae93db0f39479d1d276d06ddb9ce03a1f4346f0172be8bf23ac0 not found: ID does not exist"
Jul 08 17:24:30.345455 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.345436    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"]
Jul 08 17:24:30.349052 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:30.349030    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/router-with-refs-pd-test-kserve-57d4965598-p2stc"]
Jul 08 17:24:31.821816 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:31.821783    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" path="/var/lib/kubelet/pods/094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1/volumes"
Jul 08 17:24:31.822261 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:31.822196    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" path="/var/lib/kubelet/pods/2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce/volumes"
Jul 08 17:24:31.994756 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:31.994714    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59" podUID="33d141fe-7fff-4b18-8e2c-1d9f61bed747" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Jul 08 17:24:41.860936 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:41.860889    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:24:41.995730 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:41.995643    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59" podUID="33d141fe-7fff-4b18-8e2c-1d9f61bed747" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Jul 08 17:24:46.984626 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:46.984580    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"]
Jul 08 17:24:46.985094 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:46.985013    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" containerName="storage-initializer"
Jul 08 17:24:46.985094 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:46.985028    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" containerName="storage-initializer"
Jul 08 17:24:46.985094 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:46.985037    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerName="storage-initializer"
Jul 08 17:24:46.985094 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:46.985044    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerName="storage-initializer"
Jul 08 17:24:46.985094 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:46.985053    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerName="llm-d-routing-sidecar"
Jul 08 17:24:46.985094 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:46.985058    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerName="llm-d-routing-sidecar"
Jul 08 17:24:46.985094 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:46.985063    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerName="main"
Jul 08 17:24:46.985094 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:46.985068    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerName="main"
Jul 08 17:24:46.985094 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:46.985077    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" containerName="main"
Jul 08 17:24:46.985094 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:46.985082    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" containerName="main"
Jul 08 17:24:46.985615 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:46.985137    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerName="main"
Jul 08 17:24:46.985615 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:46.985147    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fbec2d8-b74e-4b6c-bf96-1bdcc65ad1ce" containerName="llm-d-routing-sidecar"
Jul 08 17:24:46.985615 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:46.985153    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="094a3e40-7f9e-4974-8fe5-7ab1fe58f6b1" containerName="main"
Jul 08 17:24:46.987595 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:46.987570    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"
Jul 08 17:24:46.989947 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:46.989928    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvd10b93c8eba12f87c3a350a5cae2ee0c-kserve-self-signed-certs\""
Jul 08 17:24:46.999985 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:46.999968    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"]
Jul 08 17:24:47.035706 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:47.035650    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:24:47.137100 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:47.137068    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlbnl\" (UniqueName: \"kubernetes.io/projected/a35b3855-3bf9-4451-88bc-00d60e99db9a-kube-api-access-dlbnl\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn\" (UID: \"a35b3855-3bf9-4451-88bc-00d60e99db9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"
Jul 08 17:24:47.137100 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:47.137101    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn\" (UID: \"a35b3855-3bf9-4451-88bc-00d60e99db9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"
Jul 08 17:24:47.137266 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:47.137126    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn\" (UID: \"a35b3855-3bf9-4451-88bc-00d60e99db9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"
Jul 08 17:24:47.137266 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:47.137193    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a35b3855-3bf9-4451-88bc-00d60e99db9a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn\" (UID: \"a35b3855-3bf9-4451-88bc-00d60e99db9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"
Jul 08 17:24:47.137266 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:47.137243    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn\" (UID: \"a35b3855-3bf9-4451-88bc-00d60e99db9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"
Jul 08 17:24:47.137370 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:47.137289    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn\" (UID: \"a35b3855-3bf9-4451-88bc-00d60e99db9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"
Jul 08 17:24:47.137370 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:47.137357    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn\" (UID: \"a35b3855-3bf9-4451-88bc-00d60e99db9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"
Jul 08 17:24:47.238874 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:47.238786    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn\" (UID: \"a35b3855-3bf9-4451-88bc-00d60e99db9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"
Jul 08 17:24:47.239046 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:47.238933    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlbnl\" (UniqueName: \"kubernetes.io/projected/a35b3855-3bf9-4451-88bc-00d60e99db9a-kube-api-access-dlbnl\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn\" (UID: \"a35b3855-3bf9-4451-88bc-00d60e99db9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"
Jul 08 17:24:47.239046 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:47.238962    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn\" (UID: \"a35b3855-3bf9-4451-88bc-00d60e99db9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"
Jul 08 17:24:47.239046 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:47.238997    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn\" (UID: \"a35b3855-3bf9-4451-88bc-00d60e99db9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"
Jul 08 17:24:47.239046 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:47.239028    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a35b3855-3bf9-4451-88bc-00d60e99db9a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn\" (UID: \"a35b3855-3bf9-4451-88bc-00d60e99db9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"
Jul 08 17:24:47.239277 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:47.239067    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn\" (UID: \"a35b3855-3bf9-4451-88bc-00d60e99db9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"
Jul 08 17:24:47.239277 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:47.239110    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn\" (UID: \"a35b3855-3bf9-4451-88bc-00d60e99db9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"
Jul 08 17:24:47.239277 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:47.239185    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn\" (UID: \"a35b3855-3bf9-4451-88bc-00d60e99db9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"
Jul 08 17:24:47.239441 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:47.239308    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-home\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn\" (UID: \"a35b3855-3bf9-4451-88bc-00d60e99db9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"
Jul 08 17:24:47.239500 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:47.239481    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn\" (UID: \"a35b3855-3bf9-4451-88bc-00d60e99db9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"
Jul 08 17:24:47.239558 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:47.239486    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn\" (UID: \"a35b3855-3bf9-4451-88bc-00d60e99db9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"
Jul 08 17:24:47.241296 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:47.241261    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-dshm\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn\" (UID: \"a35b3855-3bf9-4451-88bc-00d60e99db9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"
Jul 08 17:24:47.241533 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:47.241515    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a35b3855-3bf9-4451-88bc-00d60e99db9a-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn\" (UID: \"a35b3855-3bf9-4451-88bc-00d60e99db9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"
Jul 08 17:24:47.248721 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:47.248693    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlbnl\" (UniqueName: \"kubernetes.io/projected/a35b3855-3bf9-4451-88bc-00d60e99db9a-kube-api-access-dlbnl\") pod \"llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn\" (UID: \"a35b3855-3bf9-4451-88bc-00d60e99db9a\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"
Jul 08 17:24:47.299471 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:47.299437    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"
Jul 08 17:24:47.448281 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:47.448246    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"]
Jul 08 17:24:47.448619 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:24:47.448589    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda35b3855_3bf9_4451_88bc_00d60e99db9a.slice/crio-c55ad243353e290337b406c55467f1fe886503815e89e6a99147c180489af52f WatchSource:0}: Error finding container c55ad243353e290337b406c55467f1fe886503815e89e6a99147c180489af52f: Status 404 returned error can't find the container with id c55ad243353e290337b406c55467f1fe886503815e89e6a99147c180489af52f
Jul 08 17:24:48.427396 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:48.427350    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn" event={"ID":"a35b3855-3bf9-4451-88bc-00d60e99db9a","Type":"ContainerStarted","Data":"8156274583f5dee2c2de306cab418dc886ebbfdbc00821e163b92f4c0e9efebe"}
Jul 08 17:24:48.427877 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:48.427460    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn" event={"ID":"a35b3855-3bf9-4451-88bc-00d60e99db9a","Type":"ContainerStarted","Data":"c55ad243353e290337b406c55467f1fe886503815e89e6a99147c180489af52f"}
Jul 08 17:24:51.995333 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:51.995296    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59" podUID="33d141fe-7fff-4b18-8e2c-1d9f61bed747" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Jul 08 17:24:53.458024 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:53.457993    2565 generic.go:356] "Generic (PLEG): container finished" podID="a35b3855-3bf9-4451-88bc-00d60e99db9a" containerID="8156274583f5dee2c2de306cab418dc886ebbfdbc00821e163b92f4c0e9efebe" exitCode=0
Jul 08 17:24:53.458435 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:53.458081    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn" event={"ID":"a35b3855-3bf9-4451-88bc-00d60e99db9a","Type":"ContainerDied","Data":"8156274583f5dee2c2de306cab418dc886ebbfdbc00821e163b92f4c0e9efebe"}
Jul 08 17:24:54.476754 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:54.476713    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn" event={"ID":"a35b3855-3bf9-4451-88bc-00d60e99db9a","Type":"ContainerStarted","Data":"6d4b4d1037a977b0228195a2ded62b2763bdbc066f189fdf2bfc3623f025ff63"}
Jul 08 17:24:54.508374 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:54.508314    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn" podStartSLOduration=3.386196059 podStartE2EDuration="8.50829453s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.122098471s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:24:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:24:54.505457651 +0000 UTC m=+3031.218819077" watchObservedRunningTime="2026-07-08 17:24:54.50829453 +0000 UTC m=+3031.221655957"
Jul 08 17:24:55.299562 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:55.299522    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"
Jul 08 17:24:57.300254 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:57.300211    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"
Jul 08 17:24:57.301773 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:24:57.301738    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn" podUID="a35b3855-3bf9-4451-88bc-00d60e99db9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Jul 08 17:25:00.295698 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:25:00.295647    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:25:01.995605 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:25:01.995564    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59" podUID="33d141fe-7fff-4b18-8e2c-1d9f61bed747" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Jul 08 17:25:07.300528 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:25:07.300474    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn" podUID="a35b3855-3bf9-4451-88bc-00d60e99db9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Jul 08 17:25:11.995572 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:25:11.995527    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59" podUID="33d141fe-7fff-4b18-8e2c-1d9f61bed747" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Jul 08 17:25:17.300546 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:25:17.300452    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn" podUID="a35b3855-3bf9-4451-88bc-00d60e99db9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Jul 08 17:25:22.004782 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:25:22.004754    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:25:22.012459 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:25:22.012433    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:25:27.300466 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:25:27.300424    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn" podUID="a35b3855-3bf9-4451-88bc-00d60e99db9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Jul 08 17:25:37.300357 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:25:37.300314    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn" podUID="a35b3855-3bf9-4451-88bc-00d60e99db9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Jul 08 17:25:47.300739 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:25:47.300663    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn" podUID="a35b3855-3bf9-4451-88bc-00d60e99db9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Jul 08 17:25:57.300751 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:25:57.300710    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn" podUID="a35b3855-3bf9-4451-88bc-00d60e99db9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Jul 08 17:26:01.320199 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:01.320160    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:26:01.422311 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:01.422271    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:26:07.300735 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:07.300692    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn" podUID="a35b3855-3bf9-4451-88bc-00d60e99db9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Jul 08 17:26:08.233304 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:08.233246    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"]
Jul 08 17:26:08.257937 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:08.257901    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:26:17.300488 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:17.300433    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn" podUID="a35b3855-3bf9-4451-88bc-00d60e99db9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Jul 08 17:26:23.261859 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:23.261811    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59" podUID="33d141fe-7fff-4b18-8e2c-1d9f61bed747" containerName="main" containerID="cri-o://5590eed9b2605d663f3c66d21e282611053b4e6688700c71a432b12226d5e6e7" gracePeriod=45
Jul 08 17:26:24.005906 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:24.005857    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59" podUID="33d141fe-7fff-4b18-8e2c-1d9f61bed747" containerName="main" probeResult="failure" output="Get \"https://10.133.0.75:8000/health\": dial tcp 10.133.0.75:8000: connect: connection refused"
Jul 08 17:26:24.319186 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:24.319156    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:26:24.417701 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:24.417646    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkcm8\" (UniqueName: \"kubernetes.io/projected/33d141fe-7fff-4b18-8e2c-1d9f61bed747-kube-api-access-gkcm8\") pod \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") "
Jul 08 17:26:24.417906 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:24.417718    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-model-cache\") pod \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") "
Jul 08 17:26:24.417906 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:24.417763    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-dshm\") pod \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") "
Jul 08 17:26:24.417906 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:24.417784    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33d141fe-7fff-4b18-8e2c-1d9f61bed747-cabundle-cert\") pod \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") "
Jul 08 17:26:24.417906 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:24.417807    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-home\") pod \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") "
Jul 08 17:26:24.417906 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:24.417863    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-tmp-dir\") pod \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") "
Jul 08 17:26:24.417906 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:24.417903    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33d141fe-7fff-4b18-8e2c-1d9f61bed747-tls-certs\") pod \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") "
Jul 08 17:26:24.418280 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:24.417937    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-kserve-provision-location\") pod \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\" (UID: \"33d141fe-7fff-4b18-8e2c-1d9f61bed747\") "
Jul 08 17:26:24.418280 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:24.418084    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-model-cache" (OuterVolumeSpecName: "model-cache") pod "33d141fe-7fff-4b18-8e2c-1d9f61bed747" (UID: "33d141fe-7fff-4b18-8e2c-1d9f61bed747"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:26:24.418280 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:24.418138    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33d141fe-7fff-4b18-8e2c-1d9f61bed747-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "33d141fe-7fff-4b18-8e2c-1d9f61bed747" (UID: "33d141fe-7fff-4b18-8e2c-1d9f61bed747"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 08 17:26:24.418280 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:24.418192    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-home" (OuterVolumeSpecName: "home") pod "33d141fe-7fff-4b18-8e2c-1d9f61bed747" (UID: "33d141fe-7fff-4b18-8e2c-1d9f61bed747"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:26:24.418500 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:24.418311    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:26:24.418500 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:24.418331    2565 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/33d141fe-7fff-4b18-8e2c-1d9f61bed747-cabundle-cert\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:26:24.418500 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:24.418344    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:26:24.420538 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:24.420505    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33d141fe-7fff-4b18-8e2c-1d9f61bed747-kube-api-access-gkcm8" (OuterVolumeSpecName: "kube-api-access-gkcm8") pod "33d141fe-7fff-4b18-8e2c-1d9f61bed747" (UID: "33d141fe-7fff-4b18-8e2c-1d9f61bed747"). InnerVolumeSpecName "kube-api-access-gkcm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:26:24.420955 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:24.420590    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33d141fe-7fff-4b18-8e2c-1d9f61bed747-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "33d141fe-7fff-4b18-8e2c-1d9f61bed747" (UID: "33d141fe-7fff-4b18-8e2c-1d9f61bed747"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:26:24.421130 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:24.421105    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-dshm" (OuterVolumeSpecName: "dshm") pod "33d141fe-7fff-4b18-8e2c-1d9f61bed747" (UID: "33d141fe-7fff-4b18-8e2c-1d9f61bed747"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:26:24.435297 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:24.435250    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "33d141fe-7fff-4b18-8e2c-1d9f61bed747" (UID: "33d141fe-7fff-4b18-8e2c-1d9f61bed747"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:26:24.487810 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:24.487737    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "33d141fe-7fff-4b18-8e2c-1d9f61bed747" (UID: "33d141fe-7fff-4b18-8e2c-1d9f61bed747"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:26:24.519388 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:24.519354    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:26:24.519388 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:24.519381    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:26:24.519388 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:24.519391    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33d141fe-7fff-4b18-8e2c-1d9f61bed747-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:26:24.519592 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:24.519401    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/33d141fe-7fff-4b18-8e2c-1d9f61bed747-kserve-provision-location\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:26:24.519592 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:24.519410    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkcm8\" (UniqueName: \"kubernetes.io/projected/33d141fe-7fff-4b18-8e2c-1d9f61bed747-kube-api-access-gkcm8\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:26:25.073859 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:25.073812    2565 generic.go:356] "Generic (PLEG): container finished" podID="33d141fe-7fff-4b18-8e2c-1d9f61bed747" containerID="5590eed9b2605d663f3c66d21e282611053b4e6688700c71a432b12226d5e6e7" exitCode=0
Jul 08 17:26:25.074046 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:25.073883    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"
Jul 08 17:26:25.074046 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:25.073900    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59" event={"ID":"33d141fe-7fff-4b18-8e2c-1d9f61bed747","Type":"ContainerDied","Data":"5590eed9b2605d663f3c66d21e282611053b4e6688700c71a432b12226d5e6e7"}
Jul 08 17:26:25.074046 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:25.073999    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59" event={"ID":"33d141fe-7fff-4b18-8e2c-1d9f61bed747","Type":"ContainerDied","Data":"e9b8d796c4be66bf50112e93213e2d279a485a1a0d2e5bf63af7aa9ed9927ab4"}
Jul 08 17:26:25.074046 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:25.074031    2565 scope.go:117] "RemoveContainer" containerID="5590eed9b2605d663f3c66d21e282611053b4e6688700c71a432b12226d5e6e7"
Jul 08 17:26:25.084030 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:25.084007    2565 scope.go:117] "RemoveContainer" containerID="cede1513c382a80fd4a97e92cddb870468c9877b0942dd34e6e030b4d2387b2c"
Jul 08 17:26:25.092285 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:25.092260    2565 scope.go:117] "RemoveContainer" containerID="5590eed9b2605d663f3c66d21e282611053b4e6688700c71a432b12226d5e6e7"
Jul 08 17:26:25.092564 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:26:25.092542    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5590eed9b2605d663f3c66d21e282611053b4e6688700c71a432b12226d5e6e7\": container with ID starting with 5590eed9b2605d663f3c66d21e282611053b4e6688700c71a432b12226d5e6e7 not found: ID does not exist" containerID="5590eed9b2605d663f3c66d21e282611053b4e6688700c71a432b12226d5e6e7"
Jul 08 17:26:25.092645 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:25.092575    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5590eed9b2605d663f3c66d21e282611053b4e6688700c71a432b12226d5e6e7"} err="failed to get container status \"5590eed9b2605d663f3c66d21e282611053b4e6688700c71a432b12226d5e6e7\": rpc error: code = NotFound desc = could not find container \"5590eed9b2605d663f3c66d21e282611053b4e6688700c71a432b12226d5e6e7\": container with ID starting with 5590eed9b2605d663f3c66d21e282611053b4e6688700c71a432b12226d5e6e7 not found: ID does not exist"
Jul 08 17:26:25.092645 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:25.092600    2565 scope.go:117] "RemoveContainer" containerID="cede1513c382a80fd4a97e92cddb870468c9877b0942dd34e6e030b4d2387b2c"
Jul 08 17:26:25.092902 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:26:25.092883    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cede1513c382a80fd4a97e92cddb870468c9877b0942dd34e6e030b4d2387b2c\": container with ID starting with cede1513c382a80fd4a97e92cddb870468c9877b0942dd34e6e030b4d2387b2c not found: ID does not exist" containerID="cede1513c382a80fd4a97e92cddb870468c9877b0942dd34e6e030b4d2387b2c"
Jul 08 17:26:25.092947 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:25.092909    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cede1513c382a80fd4a97e92cddb870468c9877b0942dd34e6e030b4d2387b2c"} err="failed to get container status \"cede1513c382a80fd4a97e92cddb870468c9877b0942dd34e6e030b4d2387b2c\": rpc error: code = NotFound desc = could not find container \"cede1513c382a80fd4a97e92cddb870468c9877b0942dd34e6e030b4d2387b2c\": container with ID starting with cede1513c382a80fd4a97e92cddb870468c9877b0942dd34e6e030b4d2387b2c not found: ID does not exist"
Jul 08 17:26:25.099147 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:25.099125    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"]
Jul 08 17:26:25.103377 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:25.103353    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/lora-multiple-adapters-test-kserve-7c8db66c87-pzn59"]
Jul 08 17:26:25.822407 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:25.822371    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33d141fe-7fff-4b18-8e2c-1d9f61bed747" path="/var/lib/kubelet/pods/33d141fe-7fff-4b18-8e2c-1d9f61bed747/volumes"
Jul 08 17:26:27.299859 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:27.299822    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn" podUID="a35b3855-3bf9-4451-88bc-00d60e99db9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Jul 08 17:26:35.379968 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.379935    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"]
Jul 08 17:26:35.380356 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.380323    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33d141fe-7fff-4b18-8e2c-1d9f61bed747" containerName="storage-initializer"
Jul 08 17:26:35.380356 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.380334    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="33d141fe-7fff-4b18-8e2c-1d9f61bed747" containerName="storage-initializer"
Jul 08 17:26:35.380438 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.380360    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33d141fe-7fff-4b18-8e2c-1d9f61bed747" containerName="main"
Jul 08 17:26:35.380438 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.380366    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="33d141fe-7fff-4b18-8e2c-1d9f61bed747" containerName="main"
Jul 08 17:26:35.380438 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.380416    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="33d141fe-7fff-4b18-8e2c-1d9f61bed747" containerName="main"
Jul 08 17:26:35.383817 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.383800    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"
Jul 08 17:26:35.386273 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.386254    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"tls-verification-test-kserve-self-signed-certs\""
Jul 08 17:26:35.397041 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.397017    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"]
Jul 08 17:26:35.409396 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.409362    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:26:35.526217 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.526184    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps9nd\" (UniqueName: \"kubernetes.io/projected/609862c6-bab1-4917-bbdd-ff8c842b0472-kube-api-access-ps9nd\") pod \"tls-verification-test-kserve-5847b7959f-z68fl\" (UID: \"609862c6-bab1-4917-bbdd-ff8c842b0472\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"
Jul 08 17:26:35.526426 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.526225    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-model-cache\") pod \"tls-verification-test-kserve-5847b7959f-z68fl\" (UID: \"609862c6-bab1-4917-bbdd-ff8c842b0472\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"
Jul 08 17:26:35.526426 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.526318    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-dshm\") pod \"tls-verification-test-kserve-5847b7959f-z68fl\" (UID: \"609862c6-bab1-4917-bbdd-ff8c842b0472\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"
Jul 08 17:26:35.526426 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.526352    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-home\") pod \"tls-verification-test-kserve-5847b7959f-z68fl\" (UID: \"609862c6-bab1-4917-bbdd-ff8c842b0472\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"
Jul 08 17:26:35.526426 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.526414    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/609862c6-bab1-4917-bbdd-ff8c842b0472-tls-certs\") pod \"tls-verification-test-kserve-5847b7959f-z68fl\" (UID: \"609862c6-bab1-4917-bbdd-ff8c842b0472\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"
Jul 08 17:26:35.526611 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.526449    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-tmp-dir\") pod \"tls-verification-test-kserve-5847b7959f-z68fl\" (UID: \"609862c6-bab1-4917-bbdd-ff8c842b0472\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"
Jul 08 17:26:35.526611 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.526492    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-kserve-provision-location\") pod \"tls-verification-test-kserve-5847b7959f-z68fl\" (UID: \"609862c6-bab1-4917-bbdd-ff8c842b0472\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"
Jul 08 17:26:35.627210 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.627171    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-tmp-dir\") pod \"tls-verification-test-kserve-5847b7959f-z68fl\" (UID: \"609862c6-bab1-4917-bbdd-ff8c842b0472\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"
Jul 08 17:26:35.627386 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.627225    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-kserve-provision-location\") pod \"tls-verification-test-kserve-5847b7959f-z68fl\" (UID: \"609862c6-bab1-4917-bbdd-ff8c842b0472\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"
Jul 08 17:26:35.627386 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.627284    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps9nd\" (UniqueName: \"kubernetes.io/projected/609862c6-bab1-4917-bbdd-ff8c842b0472-kube-api-access-ps9nd\") pod \"tls-verification-test-kserve-5847b7959f-z68fl\" (UID: \"609862c6-bab1-4917-bbdd-ff8c842b0472\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"
Jul 08 17:26:35.627386 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.627309    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-model-cache\") pod \"tls-verification-test-kserve-5847b7959f-z68fl\" (UID: \"609862c6-bab1-4917-bbdd-ff8c842b0472\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"
Jul 08 17:26:35.627386 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.627351    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-dshm\") pod \"tls-verification-test-kserve-5847b7959f-z68fl\" (UID: \"609862c6-bab1-4917-bbdd-ff8c842b0472\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"
Jul 08 17:26:35.627595 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.627514    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-home\") pod \"tls-verification-test-kserve-5847b7959f-z68fl\" (UID: \"609862c6-bab1-4917-bbdd-ff8c842b0472\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"
Jul 08 17:26:35.627796 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.627766    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/609862c6-bab1-4917-bbdd-ff8c842b0472-tls-certs\") pod \"tls-verification-test-kserve-5847b7959f-z68fl\" (UID: \"609862c6-bab1-4917-bbdd-ff8c842b0472\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"
Jul 08 17:26:35.627924 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.627793    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-model-cache\") pod \"tls-verification-test-kserve-5847b7959f-z68fl\" (UID: \"609862c6-bab1-4917-bbdd-ff8c842b0472\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"
Jul 08 17:26:35.627924 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.627642    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-tmp-dir\") pod \"tls-verification-test-kserve-5847b7959f-z68fl\" (UID: \"609862c6-bab1-4917-bbdd-ff8c842b0472\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"
Jul 08 17:26:35.627924 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.627657    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-kserve-provision-location\") pod \"tls-verification-test-kserve-5847b7959f-z68fl\" (UID: \"609862c6-bab1-4917-bbdd-ff8c842b0472\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"
Jul 08 17:26:35.627924 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.627857    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-home\") pod \"tls-verification-test-kserve-5847b7959f-z68fl\" (UID: \"609862c6-bab1-4917-bbdd-ff8c842b0472\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"
Jul 08 17:26:35.629626 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.629600    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-dshm\") pod \"tls-verification-test-kserve-5847b7959f-z68fl\" (UID: \"609862c6-bab1-4917-bbdd-ff8c842b0472\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"
Jul 08 17:26:35.630056 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.630006    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/609862c6-bab1-4917-bbdd-ff8c842b0472-tls-certs\") pod \"tls-verification-test-kserve-5847b7959f-z68fl\" (UID: \"609862c6-bab1-4917-bbdd-ff8c842b0472\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"
Jul 08 17:26:35.641147 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.641122    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps9nd\" (UniqueName: \"kubernetes.io/projected/609862c6-bab1-4917-bbdd-ff8c842b0472-kube-api-access-ps9nd\") pod \"tls-verification-test-kserve-5847b7959f-z68fl\" (UID: \"609862c6-bab1-4917-bbdd-ff8c842b0472\") " pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"
Jul 08 17:26:35.682745 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.682697    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:26:35.693590 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.693563    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"
Jul 08 17:26:35.851138 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:35.851103    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"]
Jul 08 17:26:35.852785 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:26:35.852754    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod609862c6_bab1_4917_bbdd_ff8c842b0472.slice/crio-d3bc4d0c2cc9d5ff8bab2f63710c99f97c211d7def6e250ce5b38c4e89d96631 WatchSource:0}: Error finding container d3bc4d0c2cc9d5ff8bab2f63710c99f97c211d7def6e250ce5b38c4e89d96631: Status 404 returned error can't find the container with id d3bc4d0c2cc9d5ff8bab2f63710c99f97c211d7def6e250ce5b38c4e89d96631
Jul 08 17:26:36.139602 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:36.139499    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl" event={"ID":"609862c6-bab1-4917-bbdd-ff8c842b0472","Type":"ContainerStarted","Data":"33a1b3c490908f738f057fec9bf3c6d1ac39b1b4a123101f609a6f8efbd5bc08"}
Jul 08 17:26:36.139602 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:36.139556    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl" event={"ID":"609862c6-bab1-4917-bbdd-ff8c842b0472","Type":"ContainerStarted","Data":"d3bc4d0c2cc9d5ff8bab2f63710c99f97c211d7def6e250ce5b38c4e89d96631"}
Jul 08 17:26:37.299955 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:37.299907    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn" podUID="a35b3855-3bf9-4451-88bc-00d60e99db9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Jul 08 17:26:41.879137 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:41.879097    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:26:42.181351 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:42.181312    2565 generic.go:356] "Generic (PLEG): container finished" podID="609862c6-bab1-4917-bbdd-ff8c842b0472" containerID="33a1b3c490908f738f057fec9bf3c6d1ac39b1b4a123101f609a6f8efbd5bc08" exitCode=0
Jul 08 17:26:42.181526 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:42.181383    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl" event={"ID":"609862c6-bab1-4917-bbdd-ff8c842b0472","Type":"ContainerDied","Data":"33a1b3c490908f738f057fec9bf3c6d1ac39b1b4a123101f609a6f8efbd5bc08"}
Jul 08 17:26:43.189351 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:43.189316    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl" event={"ID":"609862c6-bab1-4917-bbdd-ff8c842b0472","Type":"ContainerStarted","Data":"b8c521bf850a56213df11d6a246fa39aec63b6b53c4ea1c5ef4b280df986f72e"}
Jul 08 17:26:43.227025 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:43.226970    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl" podStartSLOduration=2.344131875 podStartE2EDuration="8.226949041s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.882817166s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:26:43.225700343 +0000 UTC m=+3139.939061765" watchObservedRunningTime="2026-07-08 17:26:43.226949041 +0000 UTC m=+3139.940310469"
Jul 08 17:26:43.694408 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:43.694370    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"
Jul 08 17:26:45.694645 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:45.694549    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"
Jul 08 17:26:45.696709 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:45.696662    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl" podUID="609862c6-bab1-4917-bbdd-ff8c842b0472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Jul 08 17:26:47.300352 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:47.300304    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn" podUID="a35b3855-3bf9-4451-88bc-00d60e99db9a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.76:8000/health\": dial tcp 10.133.0.76:8000: connect: connection refused"
Jul 08 17:26:55.695053 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:55.695005    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl" podUID="609862c6-bab1-4917-bbdd-ff8c842b0472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Jul 08 17:26:57.309720 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:57.309690    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"
Jul 08 17:26:57.317085 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:26:57.317065    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"
Jul 08 17:27:01.386230 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:01.386184    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"]
Jul 08 17:27:01.418283 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:01.418246    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:27:05.694753 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:05.694709    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl" podUID="609862c6-bab1-4917-bbdd-ff8c842b0472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Jul 08 17:27:15.694503 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:15.694451    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl" podUID="609862c6-bab1-4917-bbdd-ff8c842b0472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Jul 08 17:27:16.409230 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:16.409181    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn" podUID="a35b3855-3bf9-4451-88bc-00d60e99db9a" containerName="main" containerID="cri-o://6d4b4d1037a977b0228195a2ded62b2763bdbc066f189fdf2bfc3623f025ff63" gracePeriod=45
Jul 08 17:27:17.084708 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.084684    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"
Jul 08 17:27:17.133993 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.133964    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a35b3855-3bf9-4451-88bc-00d60e99db9a-tls-certs\") pod \"a35b3855-3bf9-4451-88bc-00d60e99db9a\" (UID: \"a35b3855-3bf9-4451-88bc-00d60e99db9a\") "
Jul 08 17:27:17.134166 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.134095    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-dshm\") pod \"a35b3855-3bf9-4451-88bc-00d60e99db9a\" (UID: \"a35b3855-3bf9-4451-88bc-00d60e99db9a\") "
Jul 08 17:27:17.134166 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.134159    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlbnl\" (UniqueName: \"kubernetes.io/projected/a35b3855-3bf9-4451-88bc-00d60e99db9a-kube-api-access-dlbnl\") pod \"a35b3855-3bf9-4451-88bc-00d60e99db9a\" (UID: \"a35b3855-3bf9-4451-88bc-00d60e99db9a\") "
Jul 08 17:27:17.134268 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.134223    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-tmp-dir\") pod \"a35b3855-3bf9-4451-88bc-00d60e99db9a\" (UID: \"a35b3855-3bf9-4451-88bc-00d60e99db9a\") "
Jul 08 17:27:17.134325 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.134266    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-model-cache\") pod \"a35b3855-3bf9-4451-88bc-00d60e99db9a\" (UID: \"a35b3855-3bf9-4451-88bc-00d60e99db9a\") "
Jul 08 17:27:17.134824 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.134791    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-model-cache" (OuterVolumeSpecName: "model-cache") pod "a35b3855-3bf9-4451-88bc-00d60e99db9a" (UID: "a35b3855-3bf9-4451-88bc-00d60e99db9a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:27:17.136588 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.136451    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a35b3855-3bf9-4451-88bc-00d60e99db9a-kube-api-access-dlbnl" (OuterVolumeSpecName: "kube-api-access-dlbnl") pod "a35b3855-3bf9-4451-88bc-00d60e99db9a" (UID: "a35b3855-3bf9-4451-88bc-00d60e99db9a"). InnerVolumeSpecName "kube-api-access-dlbnl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:27:17.136588 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.136547    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a35b3855-3bf9-4451-88bc-00d60e99db9a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a35b3855-3bf9-4451-88bc-00d60e99db9a" (UID: "a35b3855-3bf9-4451-88bc-00d60e99db9a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:27:17.136789 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.136722    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-dshm" (OuterVolumeSpecName: "dshm") pod "a35b3855-3bf9-4451-88bc-00d60e99db9a" (UID: "a35b3855-3bf9-4451-88bc-00d60e99db9a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:27:17.148618 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.148549    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a35b3855-3bf9-4451-88bc-00d60e99db9a" (UID: "a35b3855-3bf9-4451-88bc-00d60e99db9a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:27:17.235227 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.235190    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-home\") pod \"a35b3855-3bf9-4451-88bc-00d60e99db9a\" (UID: \"a35b3855-3bf9-4451-88bc-00d60e99db9a\") "
Jul 08 17:27:17.235227 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.235233    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-kserve-provision-location\") pod \"a35b3855-3bf9-4451-88bc-00d60e99db9a\" (UID: \"a35b3855-3bf9-4451-88bc-00d60e99db9a\") "
Jul 08 17:27:17.235481 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.235401    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlbnl\" (UniqueName: \"kubernetes.io/projected/a35b3855-3bf9-4451-88bc-00d60e99db9a-kube-api-access-dlbnl\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:27:17.235481 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.235410    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:27:17.235481 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.235419    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:27:17.235481 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.235426    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a35b3855-3bf9-4451-88bc-00d60e99db9a-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:27:17.235481 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.235433    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:27:17.235890 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.235866    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-home" (OuterVolumeSpecName: "home") pod "a35b3855-3bf9-4451-88bc-00d60e99db9a" (UID: "a35b3855-3bf9-4451-88bc-00d60e99db9a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:27:17.295239 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.295185    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a35b3855-3bf9-4451-88bc-00d60e99db9a" (UID: "a35b3855-3bf9-4451-88bc-00d60e99db9a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:27:17.336562 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.336522    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:27:17.336562 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.336555    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a35b3855-3bf9-4451-88bc-00d60e99db9a-kserve-provision-location\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:27:17.391142 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.391104    2565 generic.go:356] "Generic (PLEG): container finished" podID="a35b3855-3bf9-4451-88bc-00d60e99db9a" containerID="6d4b4d1037a977b0228195a2ded62b2763bdbc066f189fdf2bfc3623f025ff63" exitCode=0
Jul 08 17:27:17.391335 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.391161    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"
Jul 08 17:27:17.391335 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.391188    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn" event={"ID":"a35b3855-3bf9-4451-88bc-00d60e99db9a","Type":"ContainerDied","Data":"6d4b4d1037a977b0228195a2ded62b2763bdbc066f189fdf2bfc3623f025ff63"}
Jul 08 17:27:17.391335 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.391246    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn" event={"ID":"a35b3855-3bf9-4451-88bc-00d60e99db9a","Type":"ContainerDied","Data":"c55ad243353e290337b406c55467f1fe886503815e89e6a99147c180489af52f"}
Jul 08 17:27:17.391335 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.391270    2565 scope.go:117] "RemoveContainer" containerID="6d4b4d1037a977b0228195a2ded62b2763bdbc066f189fdf2bfc3623f025ff63"
Jul 08 17:27:17.402464 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.402441    2565 scope.go:117] "RemoveContainer" containerID="8156274583f5dee2c2de306cab418dc886ebbfdbc00821e163b92f4c0e9efebe"
Jul 08 17:27:17.411618 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.411438    2565 scope.go:117] "RemoveContainer" containerID="6d4b4d1037a977b0228195a2ded62b2763bdbc066f189fdf2bfc3623f025ff63"
Jul 08 17:27:17.411742 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:27:17.411723    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d4b4d1037a977b0228195a2ded62b2763bdbc066f189fdf2bfc3623f025ff63\": container with ID starting with 6d4b4d1037a977b0228195a2ded62b2763bdbc066f189fdf2bfc3623f025ff63 not found: ID does not exist" containerID="6d4b4d1037a977b0228195a2ded62b2763bdbc066f189fdf2bfc3623f025ff63"
Jul 08 17:27:17.411826 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.411751    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d4b4d1037a977b0228195a2ded62b2763bdbc066f189fdf2bfc3623f025ff63"} err="failed to get container status \"6d4b4d1037a977b0228195a2ded62b2763bdbc066f189fdf2bfc3623f025ff63\": rpc error: code = NotFound desc = could not find container \"6d4b4d1037a977b0228195a2ded62b2763bdbc066f189fdf2bfc3623f025ff63\": container with ID starting with 6d4b4d1037a977b0228195a2ded62b2763bdbc066f189fdf2bfc3623f025ff63 not found: ID does not exist"
Jul 08 17:27:17.411826 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.411769    2565 scope.go:117] "RemoveContainer" containerID="8156274583f5dee2c2de306cab418dc886ebbfdbc00821e163b92f4c0e9efebe"
Jul 08 17:27:17.412074 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:27:17.412046    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8156274583f5dee2c2de306cab418dc886ebbfdbc00821e163b92f4c0e9efebe\": container with ID starting with 8156274583f5dee2c2de306cab418dc886ebbfdbc00821e163b92f4c0e9efebe not found: ID does not exist" containerID="8156274583f5dee2c2de306cab418dc886ebbfdbc00821e163b92f4c0e9efebe"
Jul 08 17:27:17.412168 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.412084    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8156274583f5dee2c2de306cab418dc886ebbfdbc00821e163b92f4c0e9efebe"} err="failed to get container status \"8156274583f5dee2c2de306cab418dc886ebbfdbc00821e163b92f4c0e9efebe\": rpc error: code = NotFound desc = could not find container \"8156274583f5dee2c2de306cab418dc886ebbfdbc00821e163b92f4c0e9efebe\": container with ID starting with 8156274583f5dee2c2de306cab418dc886ebbfdbc00821e163b92f4c0e9efebe not found: ID does not exist"
Jul 08 17:27:17.418344 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.418320    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"]
Jul 08 17:27:17.422792 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.422754    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-3c960099-kserve-6d69fc44d8842wn"]
Jul 08 17:27:17.822694 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:17.822645    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a35b3855-3bf9-4451-88bc-00d60e99db9a" path="/var/lib/kubelet/pods/a35b3855-3bf9-4451-88bc-00d60e99db9a/volumes"
Jul 08 17:27:25.694095 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:25.694053    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl" podUID="609862c6-bab1-4917-bbdd-ff8c842b0472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Jul 08 17:27:26.085036 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.084951    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:27:26.181556 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.181519    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:27:26.189374 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.189343    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 08 17:27:26.189741 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.189728    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a35b3855-3bf9-4451-88bc-00d60e99db9a" containerName="storage-initializer"
Jul 08 17:27:26.189791 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.189745    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a35b3855-3bf9-4451-88bc-00d60e99db9a" containerName="storage-initializer"
Jul 08 17:27:26.189791 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.189767    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a35b3855-3bf9-4451-88bc-00d60e99db9a" containerName="main"
Jul 08 17:27:26.189791 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.189772    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="a35b3855-3bf9-4451-88bc-00d60e99db9a" containerName="main"
Jul 08 17:27:26.189882 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.189832    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="a35b3855-3bf9-4451-88bc-00d60e99db9a" containerName="main"
Jul 08 17:27:26.192622 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.192605    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 08 17:27:26.195418 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.195396    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv32fd5468e8138ec3cd54cf7b9c18cfb0-kserve-self-signed-certs\""
Jul 08 17:27:26.195525 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.195402    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-fb-opt-125m-route-dc21cb-57cf7736-dockercfg-mwrmc\""
Jul 08 17:27:26.210148 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.210116    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 08 17:27:26.217770 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.217747    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 08 17:27:26.217892 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.217779    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 08 17:27:26.217892 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.217798    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 08 17:27:26.217892 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.217823    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 08 17:27:26.217892 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.217875    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2034fc5-0451-49ba-a1d0-c27e20f1f505-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 08 17:27:26.218042 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.217903    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk55v\" (UniqueName: \"kubernetes.io/projected/d2034fc5-0451-49ba-a1d0-c27e20f1f505-kube-api-access-fk55v\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 08 17:27:26.218042 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.217953    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 08 17:27:26.246020 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.245985    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:27:26.318925 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.318885    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 08 17:27:26.319145 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.318941    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 08 17:27:26.319145 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.318973    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 08 17:27:26.319145 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.318999    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 08 17:27:26.319145 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.319069    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2034fc5-0451-49ba-a1d0-c27e20f1f505-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 08 17:27:26.319145 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.319104    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk55v\" (UniqueName: \"kubernetes.io/projected/d2034fc5-0451-49ba-a1d0-c27e20f1f505-kube-api-access-fk55v\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 08 17:27:26.319415 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.319205    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 08 17:27:26.319415 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.319393    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-model-cache\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 08 17:27:26.319507 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.319425    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-home\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 08 17:27:26.319507 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.319477    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-kserve-provision-location\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 08 17:27:26.319580 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.319545    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-tmp-dir\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 08 17:27:26.321349 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.321327    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-dshm\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 08 17:27:26.321474 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.321455    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2034fc5-0451-49ba-a1d0-c27e20f1f505-tls-certs\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 08 17:27:26.328102 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.328080    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk55v\" (UniqueName: \"kubernetes.io/projected/d2034fc5-0451-49ba-a1d0-c27e20f1f505-kube-api-access-fk55v\") pod \"llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0\" (UID: \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\") " pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 08 17:27:26.505805 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.505759    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 08 17:27:26.543738 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.543702    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:27:26.659221 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:26.659194    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 08 17:27:26.661275 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:27:26.661245    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2034fc5_0451_49ba_a1d0_c27e20f1f505.slice/crio-bd72bb553916f567f3c4c4b7ac51278410acdd01a24381b259938c3f6356d878 WatchSource:0}: Error finding container bd72bb553916f567f3c4c4b7ac51278410acdd01a24381b259938c3f6356d878: Status 404 returned error can't find the container with id bd72bb553916f567f3c4c4b7ac51278410acdd01a24381b259938c3f6356d878
Jul 08 17:27:27.459725 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:27.459663    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"d2034fc5-0451-49ba-a1d0-c27e20f1f505","Type":"ContainerStarted","Data":"09e4b22d2f0e65fb18aa86b89e54edcc4d14857fd29b4eaff5bedbbeca8eab4e"}
Jul 08 17:27:27.460212 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:27.459787    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"d2034fc5-0451-49ba-a1d0-c27e20f1f505","Type":"ContainerStarted","Data":"bd72bb553916f567f3c4c4b7ac51278410acdd01a24381b259938c3f6356d878"}
Jul 08 17:27:32.493597 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:32.493563    2565 generic.go:356] "Generic (PLEG): container finished" podID="d2034fc5-0451-49ba-a1d0-c27e20f1f505" containerID="09e4b22d2f0e65fb18aa86b89e54edcc4d14857fd29b4eaff5bedbbeca8eab4e" exitCode=0
Jul 08 17:27:32.493986 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:32.493639    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"d2034fc5-0451-49ba-a1d0-c27e20f1f505","Type":"ContainerDied","Data":"09e4b22d2f0e65fb18aa86b89e54edcc4d14857fd29b4eaff5bedbbeca8eab4e"}
Jul 08 17:27:32.882523 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:32.882421    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:27:33.502618 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:33.502578    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"d2034fc5-0451-49ba-a1d0-c27e20f1f505","Type":"ContainerStarted","Data":"45b7776a0e2eaad2e2936bd6dd58ca47c18cb36609bf21ec453f6ba8c72b2789"}
Jul 08 17:27:33.506937 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:33.506916    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 08 17:27:33.531838 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:33.531775    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podStartSLOduration=2.384154356 podStartE2EDuration="7.531755142s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.147600786s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:27:33.530302771 +0000 UTC m=+3190.243664208" watchObservedRunningTime="2026-07-08 17:27:33.531755142 +0000 UTC m=+3190.245116569"
Jul 08 17:27:35.256339 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:35.256301    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:27:35.694402 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:35.694338    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl" podUID="609862c6-bab1-4917-bbdd-ff8c842b0472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Jul 08 17:27:36.506790 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:36.506748    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 08 17:27:36.508621 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:36.508591    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="d2034fc5-0451-49ba-a1d0-c27e20f1f505" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Jul 08 17:27:45.694379 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:45.694319    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl" podUID="609862c6-bab1-4917-bbdd-ff8c842b0472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Jul 08 17:27:46.507525 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:46.507473    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="d2034fc5-0451-49ba-a1d0-c27e20f1f505" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Jul 08 17:27:55.694163 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:55.694121    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl" podUID="609862c6-bab1-4917-bbdd-ff8c842b0472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Jul 08 17:27:56.507315 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:27:56.507265    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="d2034fc5-0451-49ba-a1d0-c27e20f1f505" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Jul 08 17:28:05.695157 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:28:05.695104    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl" podUID="609862c6-bab1-4917-bbdd-ff8c842b0472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Jul 08 17:28:06.507423 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:28:06.507376    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="d2034fc5-0451-49ba-a1d0-c27e20f1f505" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Jul 08 17:28:15.694218 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:28:15.694127    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl" podUID="609862c6-bab1-4917-bbdd-ff8c842b0472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Jul 08 17:28:16.506743 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:28:16.506692    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="d2034fc5-0451-49ba-a1d0-c27e20f1f505" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Jul 08 17:28:25.694213 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:28:25.694165    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl" podUID="609862c6-bab1-4917-bbdd-ff8c842b0472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Jul 08 17:28:26.506501 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:28:26.506461    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="d2034fc5-0451-49ba-a1d0-c27e20f1f505" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Jul 08 17:28:35.694436 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:28:35.694390    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl" podUID="609862c6-bab1-4917-bbdd-ff8c842b0472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Jul 08 17:28:36.507059 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:28:36.507018    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="d2034fc5-0451-49ba-a1d0-c27e20f1f505" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Jul 08 17:28:45.704774 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:28:45.704739    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"
Jul 08 17:28:45.712605 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:28:45.712577    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"
Jul 08 17:28:46.506588 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:28:46.506542    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="d2034fc5-0451-49ba-a1d0-c27e20f1f505" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Jul 08 17:28:54.035308 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:28:54.035274    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"]
Jul 08 17:28:54.041204 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:28:54.041072    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:28:55.231842 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:28:55.231806    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:28:56.507289 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:28:56.507244    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="d2034fc5-0451-49ba-a1d0-c27e20f1f505" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Jul 08 17:29:06.507270 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:06.507220    2565 prober.go:120] "Probe failed" probeType="Startup" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="d2034fc5-0451-49ba-a1d0-c27e20f1f505" containerName="main" probeResult="failure" output="Get \"https://10.133.0.78:8000/health\": dial tcp 10.133.0.78:8000: connect: connection refused"
Jul 08 17:29:09.062189 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:09.062125    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl" podUID="609862c6-bab1-4917-bbdd-ff8c842b0472" containerName="main" containerID="cri-o://b8c521bf850a56213df11d6a246fa39aec63b6b53c4ea1c5ef4b280df986f72e" gracePeriod=45
Jul 08 17:29:09.706217 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:09.706171    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl" podUID="609862c6-bab1-4917-bbdd-ff8c842b0472" containerName="main" probeResult="failure" output="Get \"https://10.133.0.77:8000/health\": dial tcp 10.133.0.77:8000: connect: connection refused"
Jul 08 17:29:10.105751 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:10.105718    2565 generic.go:356] "Generic (PLEG): container finished" podID="609862c6-bab1-4917-bbdd-ff8c842b0472" containerID="b8c521bf850a56213df11d6a246fa39aec63b6b53c4ea1c5ef4b280df986f72e" exitCode=0
Jul 08 17:29:10.106072 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:10.105839    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl" event={"ID":"609862c6-bab1-4917-bbdd-ff8c842b0472","Type":"ContainerDied","Data":"b8c521bf850a56213df11d6a246fa39aec63b6b53c4ea1c5ef4b280df986f72e"}
Jul 08 17:29:10.123548 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:10.123525    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"
Jul 08 17:29:10.130038 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:10.130017    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-model-cache\") pod \"609862c6-bab1-4917-bbdd-ff8c842b0472\" (UID: \"609862c6-bab1-4917-bbdd-ff8c842b0472\") "
Jul 08 17:29:10.130169 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:10.130052    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-kserve-provision-location\") pod \"609862c6-bab1-4917-bbdd-ff8c842b0472\" (UID: \"609862c6-bab1-4917-bbdd-ff8c842b0472\") "
Jul 08 17:29:10.130169 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:10.130077    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-dshm\") pod \"609862c6-bab1-4917-bbdd-ff8c842b0472\" (UID: \"609862c6-bab1-4917-bbdd-ff8c842b0472\") "
Jul 08 17:29:10.130169 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:10.130133    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/609862c6-bab1-4917-bbdd-ff8c842b0472-tls-certs\") pod \"609862c6-bab1-4917-bbdd-ff8c842b0472\" (UID: \"609862c6-bab1-4917-bbdd-ff8c842b0472\") "
Jul 08 17:29:10.130169 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:10.130158    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-tmp-dir\") pod \"609862c6-bab1-4917-bbdd-ff8c842b0472\" (UID: \"609862c6-bab1-4917-bbdd-ff8c842b0472\") "
Jul 08 17:29:10.130441 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:10.130185    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ps9nd\" (UniqueName: \"kubernetes.io/projected/609862c6-bab1-4917-bbdd-ff8c842b0472-kube-api-access-ps9nd\") pod \"609862c6-bab1-4917-bbdd-ff8c842b0472\" (UID: \"609862c6-bab1-4917-bbdd-ff8c842b0472\") "
Jul 08 17:29:10.130441 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:10.130233    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-home\") pod \"609862c6-bab1-4917-bbdd-ff8c842b0472\" (UID: \"609862c6-bab1-4917-bbdd-ff8c842b0472\") "
Jul 08 17:29:10.130441 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:10.130324    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-model-cache" (OuterVolumeSpecName: "model-cache") pod "609862c6-bab1-4917-bbdd-ff8c842b0472" (UID: "609862c6-bab1-4917-bbdd-ff8c842b0472"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:29:10.130598 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:10.130531    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:29:10.131130 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:10.131108    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-home" (OuterVolumeSpecName: "home") pod "609862c6-bab1-4917-bbdd-ff8c842b0472" (UID: "609862c6-bab1-4917-bbdd-ff8c842b0472"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:29:10.133031 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:10.132981    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-dshm" (OuterVolumeSpecName: "dshm") pod "609862c6-bab1-4917-bbdd-ff8c842b0472" (UID: "609862c6-bab1-4917-bbdd-ff8c842b0472"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:29:10.133031 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:10.132988    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/609862c6-bab1-4917-bbdd-ff8c842b0472-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "609862c6-bab1-4917-bbdd-ff8c842b0472" (UID: "609862c6-bab1-4917-bbdd-ff8c842b0472"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:29:10.133031 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:10.133002    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/609862c6-bab1-4917-bbdd-ff8c842b0472-kube-api-access-ps9nd" (OuterVolumeSpecName: "kube-api-access-ps9nd") pod "609862c6-bab1-4917-bbdd-ff8c842b0472" (UID: "609862c6-bab1-4917-bbdd-ff8c842b0472"). InnerVolumeSpecName "kube-api-access-ps9nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:29:10.148235 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:10.148209    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "609862c6-bab1-4917-bbdd-ff8c842b0472" (UID: "609862c6-bab1-4917-bbdd-ff8c842b0472"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:29:10.216086 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:10.216040    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "609862c6-bab1-4917-bbdd-ff8c842b0472" (UID: "609862c6-bab1-4917-bbdd-ff8c842b0472"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:29:10.231329 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:10.231257    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:29:10.231329 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:10.231287    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-kserve-provision-location\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:29:10.231329 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:10.231297    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:29:10.231329 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:10.231305    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/609862c6-bab1-4917-bbdd-ff8c842b0472-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:29:10.231329 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:10.231317    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/609862c6-bab1-4917-bbdd-ff8c842b0472-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:29:10.231329 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:10.231326    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps9nd\" (UniqueName: \"kubernetes.io/projected/609862c6-bab1-4917-bbdd-ff8c842b0472-kube-api-access-ps9nd\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:29:11.113188 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:11.113162    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"
Jul 08 17:29:11.113626 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:11.113185    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl" event={"ID":"609862c6-bab1-4917-bbdd-ff8c842b0472","Type":"ContainerDied","Data":"d3bc4d0c2cc9d5ff8bab2f63710c99f97c211d7def6e250ce5b38c4e89d96631"}
Jul 08 17:29:11.113626 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:11.113244    2565 scope.go:117] "RemoveContainer" containerID="b8c521bf850a56213df11d6a246fa39aec63b6b53c4ea1c5ef4b280df986f72e"
Jul 08 17:29:11.122698 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:11.122660    2565 scope.go:117] "RemoveContainer" containerID="33a1b3c490908f738f057fec9bf3c6d1ac39b1b4a123101f609a6f8efbd5bc08"
Jul 08 17:29:11.143094 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:11.143067    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"]
Jul 08 17:29:11.148947 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:11.148920    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/tls-verification-test-kserve-5847b7959f-z68fl"]
Jul 08 17:29:11.828719 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:11.828652    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="609862c6-bab1-4917-bbdd-ff8c842b0472" path="/var/lib/kubelet/pods/609862c6-bab1-4917-bbdd-ff8c842b0472/volumes"
Jul 08 17:29:16.516025 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:16.515994    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 08 17:29:16.523463 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:16.523438    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 08 17:29:20.017889 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:20.017858    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:29:20.247225 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:20.247195    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj"]
Jul 08 17:29:20.247570 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:20.247559    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="609862c6-bab1-4917-bbdd-ff8c842b0472" containerName="main"
Jul 08 17:29:20.247614 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:20.247571    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="609862c6-bab1-4917-bbdd-ff8c842b0472" containerName="main"
Jul 08 17:29:20.247614 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:20.247586    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="609862c6-bab1-4917-bbdd-ff8c842b0472" containerName="storage-initializer"
Jul 08 17:29:20.247614 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:20.247592    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="609862c6-bab1-4917-bbdd-ff8c842b0472" containerName="storage-initializer"
Jul 08 17:29:20.247734 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:20.247690    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="609862c6-bab1-4917-bbdd-ff8c842b0472" containerName="main"
Jul 08 17:29:20.252693 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:20.252661    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj"
Jul 08 17:29:20.255168 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:20.255144    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-epp-sa-dockercfg-xw4zv\""
Jul 08 17:29:20.255356 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:20.255339    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"prestop-hook-test-kserve-self-signed-certs\""
Jul 08 17:29:20.260902 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:20.260881    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj"]
Jul 08 17:29:20.268148 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:20.268050    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:29:20.325087 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:20.325059    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/072e12d8-adce-4dc3-973a-eb218dd457d1-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj\" (UID: \"072e12d8-adce-4dc3-973a-eb218dd457d1\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj"
Jul 08 17:29:20.325235 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:20.325096    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr7vn\" (UniqueName: \"kubernetes.io/projected/072e12d8-adce-4dc3-973a-eb218dd457d1-kube-api-access-wr7vn\") pod \"prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj\" (UID: \"072e12d8-adce-4dc3-973a-eb218dd457d1\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj"
Jul 08 17:29:20.325235 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:20.325179    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/072e12d8-adce-4dc3-973a-eb218dd457d1-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj\" (UID: \"072e12d8-adce-4dc3-973a-eb218dd457d1\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj"
Jul 08 17:29:20.325349 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:20.325259    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/072e12d8-adce-4dc3-973a-eb218dd457d1-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj\" (UID: \"072e12d8-adce-4dc3-973a-eb218dd457d1\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj"
Jul 08 17:29:20.426473 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:20.426438    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/072e12d8-adce-4dc3-973a-eb218dd457d1-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj\" (UID: \"072e12d8-adce-4dc3-973a-eb218dd457d1\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj"
Jul 08 17:29:20.426636 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:20.426490    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr7vn\" (UniqueName: \"kubernetes.io/projected/072e12d8-adce-4dc3-973a-eb218dd457d1-kube-api-access-wr7vn\") pod \"prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj\" (UID: \"072e12d8-adce-4dc3-973a-eb218dd457d1\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj"
Jul 08 17:29:20.426636 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:20.426559    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/072e12d8-adce-4dc3-973a-eb218dd457d1-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj\" (UID: \"072e12d8-adce-4dc3-973a-eb218dd457d1\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj"
Jul 08 17:29:20.426746 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:20.426638    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/072e12d8-adce-4dc3-973a-eb218dd457d1-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj\" (UID: \"072e12d8-adce-4dc3-973a-eb218dd457d1\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj"
Jul 08 17:29:20.426930 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:20.426907    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/072e12d8-adce-4dc3-973a-eb218dd457d1-tokenizer-uds\") pod \"prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj\" (UID: \"072e12d8-adce-4dc3-973a-eb218dd457d1\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj"
Jul 08 17:29:20.426974 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:20.426953    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/072e12d8-adce-4dc3-973a-eb218dd457d1-kserve-provision-location\") pod \"prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj\" (UID: \"072e12d8-adce-4dc3-973a-eb218dd457d1\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj"
Jul 08 17:29:20.429079 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:20.429052    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/072e12d8-adce-4dc3-973a-eb218dd457d1-tls-certs\") pod \"prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj\" (UID: \"072e12d8-adce-4dc3-973a-eb218dd457d1\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj"
Jul 08 17:29:20.440432 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:20.440406    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr7vn\" (UniqueName: \"kubernetes.io/projected/072e12d8-adce-4dc3-973a-eb218dd457d1-kube-api-access-wr7vn\") pod \"prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj\" (UID: \"072e12d8-adce-4dc3-973a-eb218dd457d1\") " pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj"
Jul 08 17:29:20.562834 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:20.562754    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj"
Jul 08 17:29:20.699441 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:20.699413    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj"]
Jul 08 17:29:20.700144 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:29:20.700108    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod072e12d8_adce_4dc3_973a_eb218dd457d1.slice/crio-366cd226682a802c62682fd877dd1c738f1c99373421e3330a9aa2c0906fae22 WatchSource:0}: Error finding container 366cd226682a802c62682fd877dd1c738f1c99373421e3330a9aa2c0906fae22: Status 404 returned error can't find the container with id 366cd226682a802c62682fd877dd1c738f1c99373421e3330a9aa2c0906fae22
Jul 08 17:29:21.172362 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:21.172331    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj" event={"ID":"072e12d8-adce-4dc3-973a-eb218dd457d1","Type":"ContainerStarted","Data":"63ea31b40e3b7f07a081dd6edeabcf1e1634ef632b0fbc9a75b46e974910cd82"}
Jul 08 17:29:21.172839 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:21.172391    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj" event={"ID":"072e12d8-adce-4dc3-973a-eb218dd457d1","Type":"ContainerStarted","Data":"366cd226682a802c62682fd877dd1c738f1c99373421e3330a9aa2c0906fae22"}
Jul 08 17:29:22.179321 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:22.179291    2565 generic.go:356] "Generic (PLEG): container finished" podID="072e12d8-adce-4dc3-973a-eb218dd457d1" containerID="63ea31b40e3b7f07a081dd6edeabcf1e1634ef632b0fbc9a75b46e974910cd82" exitCode=0
Jul 08 17:29:22.179707 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:22.179378    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj" event={"ID":"072e12d8-adce-4dc3-973a-eb218dd457d1","Type":"ContainerDied","Data":"63ea31b40e3b7f07a081dd6edeabcf1e1634ef632b0fbc9a75b46e974910cd82"}
Jul 08 17:29:23.187157 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:23.187125    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj" event={"ID":"072e12d8-adce-4dc3-973a-eb218dd457d1","Type":"ContainerStarted","Data":"ec73f15fd1290add965f279f9b4da29063c17914b639d9a992eeaa1854c6cd37"}
Jul 08 17:29:23.187556 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:23.187226    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj"
Jul 08 17:29:23.209640 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:23.209585    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj" podStartSLOduration=2.534176032 podStartE2EDuration="3.209569072s" totalImagesPullingTime="0s" totalInitContainerRuntime="675.39304ms" isStatefulPod=false podCreationTimestamp="2026-07-08 17:29:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:29:23.207089592 +0000 UTC m=+3299.920451035" watchObservedRunningTime="2026-07-08 17:29:23.209569072 +0000 UTC m=+3299.922930496"
Jul 08 17:29:28.371854 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:28.371806    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:29:30.389479 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:30.389445    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:29:30.410670 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:30.410637    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 08 17:29:30.453850 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:30.453812    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:29:45.435663 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:45.435552    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" podUID="d2034fc5-0451-49ba-a1d0-c27e20f1f505" containerName="main" containerID="cri-o://45b7776a0e2eaad2e2936bd6dd58ca47c18cb36609bf21ec453f6ba8c72b2789" gracePeriod=45
Jul 08 17:29:46.180884 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.180856    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 08 17:29:46.276301 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.276212    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-tmp-dir\") pod \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\" (UID: \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\") "
Jul 08 17:29:46.276301 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.276300    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-model-cache\") pod \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\" (UID: \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\") "
Jul 08 17:29:46.276522 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.276321    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-kserve-provision-location\") pod \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\" (UID: \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\") "
Jul 08 17:29:46.276522 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.276342    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fk55v\" (UniqueName: \"kubernetes.io/projected/d2034fc5-0451-49ba-a1d0-c27e20f1f505-kube-api-access-fk55v\") pod \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\" (UID: \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\") "
Jul 08 17:29:46.276522 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.276363    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-home\") pod \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\" (UID: \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\") "
Jul 08 17:29:46.276522 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.276409    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-dshm\") pod \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\" (UID: \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\") "
Jul 08 17:29:46.276522 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.276448    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2034fc5-0451-49ba-a1d0-c27e20f1f505-tls-certs\") pod \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\" (UID: \"d2034fc5-0451-49ba-a1d0-c27e20f1f505\") "
Jul 08 17:29:46.276784 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.276631    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-model-cache" (OuterVolumeSpecName: "model-cache") pod "d2034fc5-0451-49ba-a1d0-c27e20f1f505" (UID: "d2034fc5-0451-49ba-a1d0-c27e20f1f505"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:29:46.276784 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.276775    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:29:46.277148 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.277120    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-home" (OuterVolumeSpecName: "home") pod "d2034fc5-0451-49ba-a1d0-c27e20f1f505" (UID: "d2034fc5-0451-49ba-a1d0-c27e20f1f505"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:29:46.278879 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.278669    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-dshm" (OuterVolumeSpecName: "dshm") pod "d2034fc5-0451-49ba-a1d0-c27e20f1f505" (UID: "d2034fc5-0451-49ba-a1d0-c27e20f1f505"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:29:46.278879 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.278699    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2034fc5-0451-49ba-a1d0-c27e20f1f505-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d2034fc5-0451-49ba-a1d0-c27e20f1f505" (UID: "d2034fc5-0451-49ba-a1d0-c27e20f1f505"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:29:46.278879 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.278858    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2034fc5-0451-49ba-a1d0-c27e20f1f505-kube-api-access-fk55v" (OuterVolumeSpecName: "kube-api-access-fk55v") pod "d2034fc5-0451-49ba-a1d0-c27e20f1f505" (UID: "d2034fc5-0451-49ba-a1d0-c27e20f1f505"). InnerVolumeSpecName "kube-api-access-fk55v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:29:46.289312 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.289287    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d2034fc5-0451-49ba-a1d0-c27e20f1f505" (UID: "d2034fc5-0451-49ba-a1d0-c27e20f1f505"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:29:46.319851 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.319818    2565 generic.go:356] "Generic (PLEG): container finished" podID="d2034fc5-0451-49ba-a1d0-c27e20f1f505" containerID="45b7776a0e2eaad2e2936bd6dd58ca47c18cb36609bf21ec453f6ba8c72b2789" exitCode=0
Jul 08 17:29:46.320006 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.319879    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"
Jul 08 17:29:46.320006 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.319904    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"d2034fc5-0451-49ba-a1d0-c27e20f1f505","Type":"ContainerDied","Data":"45b7776a0e2eaad2e2936bd6dd58ca47c18cb36609bf21ec453f6ba8c72b2789"}
Jul 08 17:29:46.320006 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.319965    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0" event={"ID":"d2034fc5-0451-49ba-a1d0-c27e20f1f505","Type":"ContainerDied","Data":"bd72bb553916f567f3c4c4b7ac51278410acdd01a24381b259938c3f6356d878"}
Jul 08 17:29:46.320006 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.319988    2565 scope.go:117] "RemoveContainer" containerID="45b7776a0e2eaad2e2936bd6dd58ca47c18cb36609bf21ec453f6ba8c72b2789"
Jul 08 17:29:46.329768 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.329748    2565 scope.go:117] "RemoveContainer" containerID="09e4b22d2f0e65fb18aa86b89e54edcc4d14857fd29b4eaff5bedbbeca8eab4e"
Jul 08 17:29:46.337867 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.337848    2565 scope.go:117] "RemoveContainer" containerID="45b7776a0e2eaad2e2936bd6dd58ca47c18cb36609bf21ec453f6ba8c72b2789"
Jul 08 17:29:46.338164 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:29:46.338142    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45b7776a0e2eaad2e2936bd6dd58ca47c18cb36609bf21ec453f6ba8c72b2789\": container with ID starting with 45b7776a0e2eaad2e2936bd6dd58ca47c18cb36609bf21ec453f6ba8c72b2789 not found: ID does not exist" containerID="45b7776a0e2eaad2e2936bd6dd58ca47c18cb36609bf21ec453f6ba8c72b2789"
Jul 08 17:29:46.338261 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.338170    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45b7776a0e2eaad2e2936bd6dd58ca47c18cb36609bf21ec453f6ba8c72b2789"} err="failed to get container status \"45b7776a0e2eaad2e2936bd6dd58ca47c18cb36609bf21ec453f6ba8c72b2789\": rpc error: code = NotFound desc = could not find container \"45b7776a0e2eaad2e2936bd6dd58ca47c18cb36609bf21ec453f6ba8c72b2789\": container with ID starting with 45b7776a0e2eaad2e2936bd6dd58ca47c18cb36609bf21ec453f6ba8c72b2789 not found: ID does not exist"
Jul 08 17:29:46.338261 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.338189    2565 scope.go:117] "RemoveContainer" containerID="09e4b22d2f0e65fb18aa86b89e54edcc4d14857fd29b4eaff5bedbbeca8eab4e"
Jul 08 17:29:46.338474 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:29:46.338456    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09e4b22d2f0e65fb18aa86b89e54edcc4d14857fd29b4eaff5bedbbeca8eab4e\": container with ID starting with 09e4b22d2f0e65fb18aa86b89e54edcc4d14857fd29b4eaff5bedbbeca8eab4e not found: ID does not exist" containerID="09e4b22d2f0e65fb18aa86b89e54edcc4d14857fd29b4eaff5bedbbeca8eab4e"
Jul 08 17:29:46.338518 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.338481    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09e4b22d2f0e65fb18aa86b89e54edcc4d14857fd29b4eaff5bedbbeca8eab4e"} err="failed to get container status \"09e4b22d2f0e65fb18aa86b89e54edcc4d14857fd29b4eaff5bedbbeca8eab4e\": rpc error: code = NotFound desc = could not find container \"09e4b22d2f0e65fb18aa86b89e54edcc4d14857fd29b4eaff5bedbbeca8eab4e\": container with ID starting with 09e4b22d2f0e65fb18aa86b89e54edcc4d14857fd29b4eaff5bedbbeca8eab4e not found: ID does not exist"
Jul 08 17:29:46.343291 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.343270    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d2034fc5-0451-49ba-a1d0-c27e20f1f505" (UID: "d2034fc5-0451-49ba-a1d0-c27e20f1f505"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:29:46.378047 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.378020    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:29:46.378047 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.378046    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-kserve-provision-location\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:29:46.378181 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.378055    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk55v\" (UniqueName: \"kubernetes.io/projected/d2034fc5-0451-49ba-a1d0-c27e20f1f505-kube-api-access-fk55v\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:29:46.378181 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.378066    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:29:46.378181 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.378074    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2034fc5-0451-49ba-a1d0-c27e20f1f505-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:29:46.378181 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.378081    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2034fc5-0451-49ba-a1d0-c27e20f1f505-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:29:46.644836 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.644805    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 08 17:29:46.649212 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:46.649187    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc-model-fb-opt-125m-route-dc21cb14-kserve-mn-0"]
Jul 08 17:29:47.821787 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:47.821748    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2034fc5-0451-49ba-a1d0-c27e20f1f505" path="/var/lib/kubelet/pods/d2034fc5-0451-49ba-a1d0-c27e20f1f505/volumes"
Jul 08 17:29:54.194068 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:29:54.194037    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj"
Jul 08 17:30:26.420568 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:30:26.420518    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:30:26.488737 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:30:26.488696    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:30:43.561530 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:30:43.561494    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:30:59.289140 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:30:59.289099    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:31:00.018586 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:00.018547    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:31:37.308121 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:37.308085    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl"]
Jul 08 17:31:37.308578 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:37.308498    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2034fc5-0451-49ba-a1d0-c27e20f1f505" containerName="main"
Jul 08 17:31:37.308578 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:37.308511    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2034fc5-0451-49ba-a1d0-c27e20f1f505" containerName="main"
Jul 08 17:31:37.308578 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:37.308522    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2034fc5-0451-49ba-a1d0-c27e20f1f505" containerName="storage-initializer"
Jul 08 17:31:37.308578 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:37.308528    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2034fc5-0451-49ba-a1d0-c27e20f1f505" containerName="storage-initializer"
Jul 08 17:31:37.308739 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:37.308591    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2034fc5-0451-49ba-a1d0-c27e20f1f505" containerName="main"
Jul 08 17:31:37.311751 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:37.311734    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl"
Jul 08 17:31:37.315465 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:37.315441    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisvc-model-qwen2-5-0-5b-rout-4c1ba212-epp-sa-dockercfg-hgwxd\""
Jul 08 17:31:37.315608 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:37.315445    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"llmisv431d7e30f13554296364403a5b6365f5-kserve-self-signed-certs\""
Jul 08 17:31:37.322597 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:37.322566    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl"]
Jul 08 17:31:37.324008 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:37.323978    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:31:37.392176 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:37.392136    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8b659b11-9b1c-4d88-8e76-5144b0c43758-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl\" (UID: \"8b659b11-9b1c-4d88-8e76-5144b0c43758\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl"
Jul 08 17:31:37.392340 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:37.392212    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b659b11-9b1c-4d88-8e76-5144b0c43758-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl\" (UID: \"8b659b11-9b1c-4d88-8e76-5144b0c43758\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl"
Jul 08 17:31:37.392340 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:37.392280    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwmx6\" (UniqueName: \"kubernetes.io/projected/8b659b11-9b1c-4d88-8e76-5144b0c43758-kube-api-access-zwmx6\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl\" (UID: \"8b659b11-9b1c-4d88-8e76-5144b0c43758\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl"
Jul 08 17:31:37.423803 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:37.423767    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:31:37.493470 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:37.493440    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwmx6\" (UniqueName: \"kubernetes.io/projected/8b659b11-9b1c-4d88-8e76-5144b0c43758-kube-api-access-zwmx6\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl\" (UID: \"8b659b11-9b1c-4d88-8e76-5144b0c43758\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl"
Jul 08 17:31:37.493661 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:37.493491    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8b659b11-9b1c-4d88-8e76-5144b0c43758-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl\" (UID: \"8b659b11-9b1c-4d88-8e76-5144b0c43758\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl"
Jul 08 17:31:37.493661 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:37.493551    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b659b11-9b1c-4d88-8e76-5144b0c43758-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl\" (UID: \"8b659b11-9b1c-4d88-8e76-5144b0c43758\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl"
Jul 08 17:31:37.493912 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:37.493888    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8b659b11-9b1c-4d88-8e76-5144b0c43758-tokenizer-uds\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl\" (UID: \"8b659b11-9b1c-4d88-8e76-5144b0c43758\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl"
Jul 08 17:31:37.496088 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:37.496056    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b659b11-9b1c-4d88-8e76-5144b0c43758-tls-certs\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl\" (UID: \"8b659b11-9b1c-4d88-8e76-5144b0c43758\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl"
Jul 08 17:31:37.502440 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:37.502417    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwmx6\" (UniqueName: \"kubernetes.io/projected/8b659b11-9b1c-4d88-8e76-5144b0c43758-kube-api-access-zwmx6\") pod \"llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl\" (UID: \"8b659b11-9b1c-4d88-8e76-5144b0c43758\") " pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl"
Jul 08 17:31:37.622653 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:37.622567    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl"
Jul 08 17:31:37.765721 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:37.763693    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl"]
Jul 08 17:31:37.770956 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:31:37.770909    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b659b11_9b1c_4d88_8e76_5144b0c43758.slice/crio-586216eb990c7d667465f680dec5e22b8933e2b494bf4a0e6349808cca935f68 WatchSource:0}: Error finding container 586216eb990c7d667465f680dec5e22b8933e2b494bf4a0e6349808cca935f68: Status 404 returned error can't find the container with id 586216eb990c7d667465f680dec5e22b8933e2b494bf4a0e6349808cca935f68
Jul 08 17:31:37.940618 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:37.940578    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl" event={"ID":"8b659b11-9b1c-4d88-8e76-5144b0c43758","Type":"ContainerStarted","Data":"3532dc48786e5c8a0c67e415751a1bc017ac39f330a5402efc5b3e31421d00ad"}
Jul 08 17:31:37.940805 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:37.940637    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl" event={"ID":"8b659b11-9b1c-4d88-8e76-5144b0c43758","Type":"ContainerStarted","Data":"586216eb990c7d667465f680dec5e22b8933e2b494bf4a0e6349808cca935f68"}
Jul 08 17:31:37.940805 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:37.940717    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl"
Jul 08 17:31:37.966866 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:37.966809    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl" podStartSLOduration=0.966789834 podStartE2EDuration="966.789834ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:31:37.964122292 +0000 UTC m=+3434.677483718" watchObservedRunningTime="2026-07-08 17:31:37.966789834 +0000 UTC m=+3434.680151261"
Jul 08 17:31:53.925601 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:53.925569    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:31:55.443814 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:55.443776    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:31:55.458910 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:55.458881    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj"]
Jul 08 17:31:55.469716 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:31:55.469668    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 17:31:55.469716 ip-10-0-128-142 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:31:55.469716 ip-10-0-128-142 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 08 17:31:55.469716 ip-10-0-128-142 kubenswrapper[2565]:  > containerID="ec73f15fd1290add965f279f9b4da29063c17914b639d9a992eeaa1854c6cd37" cmd=["/bin/sleep","15"]
Jul 08 17:31:55.470026 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:55.469730    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj" message="" err=<
Jul 08 17:31:55.470026 ip-10-0-128-142 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:31:55.470026 ip-10-0-128-142 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 08 17:31:55.470026 ip-10-0-128-142 kubenswrapper[2565]:  >
Jul 08 17:31:55.470026 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:31:55.469743    2565 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 17:31:55.470026 ip-10-0-128-142 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:31:55.470026 ip-10-0-128-142 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 08 17:31:55.470026 ip-10-0-128-142 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj" podUID="072e12d8-adce-4dc3-973a-eb218dd457d1" containerName="main" containerID="cri-o://ec73f15fd1290add965f279f9b4da29063c17914b639d9a992eeaa1854c6cd37"
Jul 08 17:31:55.470026 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:55.469779    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj" podUID="072e12d8-adce-4dc3-973a-eb218dd457d1" containerName="main" containerID="cri-o://ec73f15fd1290add965f279f9b4da29063c17914b639d9a992eeaa1854c6cd37" gracePeriod=60
Jul 08 17:31:55.522632 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:55.522606    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:31:55.723108 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:55.723081    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj"
Jul 08 17:31:55.852167 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:55.852141    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/072e12d8-adce-4dc3-973a-eb218dd457d1-kserve-provision-location\") pod \"072e12d8-adce-4dc3-973a-eb218dd457d1\" (UID: \"072e12d8-adce-4dc3-973a-eb218dd457d1\") "
Jul 08 17:31:55.852310 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:55.852204    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/072e12d8-adce-4dc3-973a-eb218dd457d1-tls-certs\") pod \"072e12d8-adce-4dc3-973a-eb218dd457d1\" (UID: \"072e12d8-adce-4dc3-973a-eb218dd457d1\") "
Jul 08 17:31:55.852310 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:55.852233    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/072e12d8-adce-4dc3-973a-eb218dd457d1-tokenizer-uds\") pod \"072e12d8-adce-4dc3-973a-eb218dd457d1\" (UID: \"072e12d8-adce-4dc3-973a-eb218dd457d1\") "
Jul 08 17:31:55.852310 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:55.852280    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr7vn\" (UniqueName: \"kubernetes.io/projected/072e12d8-adce-4dc3-973a-eb218dd457d1-kube-api-access-wr7vn\") pod \"072e12d8-adce-4dc3-973a-eb218dd457d1\" (UID: \"072e12d8-adce-4dc3-973a-eb218dd457d1\") "
Jul 08 17:31:55.852508 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:55.852487    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/072e12d8-adce-4dc3-973a-eb218dd457d1-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "072e12d8-adce-4dc3-973a-eb218dd457d1" (UID: "072e12d8-adce-4dc3-973a-eb218dd457d1"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:31:55.852943 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:55.852922    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/072e12d8-adce-4dc3-973a-eb218dd457d1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "072e12d8-adce-4dc3-973a-eb218dd457d1" (UID: "072e12d8-adce-4dc3-973a-eb218dd457d1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:31:55.854125 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:55.854103    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/072e12d8-adce-4dc3-973a-eb218dd457d1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "072e12d8-adce-4dc3-973a-eb218dd457d1" (UID: "072e12d8-adce-4dc3-973a-eb218dd457d1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:31:55.854251 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:55.854230    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/072e12d8-adce-4dc3-973a-eb218dd457d1-kube-api-access-wr7vn" (OuterVolumeSpecName: "kube-api-access-wr7vn") pod "072e12d8-adce-4dc3-973a-eb218dd457d1" (UID: "072e12d8-adce-4dc3-973a-eb218dd457d1"). InnerVolumeSpecName "kube-api-access-wr7vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:31:55.953375 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:55.953313    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/072e12d8-adce-4dc3-973a-eb218dd457d1-tokenizer-uds\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:31:55.953375 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:55.953336    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr7vn\" (UniqueName: \"kubernetes.io/projected/072e12d8-adce-4dc3-973a-eb218dd457d1-kube-api-access-wr7vn\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:31:55.953544 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:55.953421    2565 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/072e12d8-adce-4dc3-973a-eb218dd457d1-kserve-provision-location\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:31:55.953544 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:55.953516    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/072e12d8-adce-4dc3-973a-eb218dd457d1-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:31:56.040631 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:56.040605    2565 generic.go:356] "Generic (PLEG): container finished" podID="072e12d8-adce-4dc3-973a-eb218dd457d1" containerID="ec73f15fd1290add965f279f9b4da29063c17914b639d9a992eeaa1854c6cd37" exitCode=0
Jul 08 17:31:56.040755 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:56.040646    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj" event={"ID":"072e12d8-adce-4dc3-973a-eb218dd457d1","Type":"ContainerDied","Data":"ec73f15fd1290add965f279f9b4da29063c17914b639d9a992eeaa1854c6cd37"}
Jul 08 17:31:56.040755 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:56.040654    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj"
Jul 08 17:31:56.040755 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:56.040696    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj" event={"ID":"072e12d8-adce-4dc3-973a-eb218dd457d1","Type":"ContainerDied","Data":"366cd226682a802c62682fd877dd1c738f1c99373421e3330a9aa2c0906fae22"}
Jul 08 17:31:56.040755 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:56.040718    2565 scope.go:117] "RemoveContainer" containerID="ec73f15fd1290add965f279f9b4da29063c17914b639d9a992eeaa1854c6cd37"
Jul 08 17:31:56.050201 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:56.050183    2565 scope.go:117] "RemoveContainer" containerID="63ea31b40e3b7f07a081dd6edeabcf1e1634ef632b0fbc9a75b46e974910cd82"
Jul 08 17:31:56.060316 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:56.060298    2565 scope.go:117] "RemoveContainer" containerID="ec73f15fd1290add965f279f9b4da29063c17914b639d9a992eeaa1854c6cd37"
Jul 08 17:31:56.060601 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:31:56.060580    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec73f15fd1290add965f279f9b4da29063c17914b639d9a992eeaa1854c6cd37\": container with ID starting with ec73f15fd1290add965f279f9b4da29063c17914b639d9a992eeaa1854c6cd37 not found: ID does not exist" containerID="ec73f15fd1290add965f279f9b4da29063c17914b639d9a992eeaa1854c6cd37"
Jul 08 17:31:56.060688 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:56.060609    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec73f15fd1290add965f279f9b4da29063c17914b639d9a992eeaa1854c6cd37"} err="failed to get container status \"ec73f15fd1290add965f279f9b4da29063c17914b639d9a992eeaa1854c6cd37\": rpc error: code = NotFound desc = could not find container \"ec73f15fd1290add965f279f9b4da29063c17914b639d9a992eeaa1854c6cd37\": container with ID starting with ec73f15fd1290add965f279f9b4da29063c17914b639d9a992eeaa1854c6cd37 not found: ID does not exist"
Jul 08 17:31:56.060688 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:56.060627    2565 scope.go:117] "RemoveContainer" containerID="63ea31b40e3b7f07a081dd6edeabcf1e1634ef632b0fbc9a75b46e974910cd82"
Jul 08 17:31:56.060924 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:31:56.060902    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63ea31b40e3b7f07a081dd6edeabcf1e1634ef632b0fbc9a75b46e974910cd82\": container with ID starting with 63ea31b40e3b7f07a081dd6edeabcf1e1634ef632b0fbc9a75b46e974910cd82 not found: ID does not exist" containerID="63ea31b40e3b7f07a081dd6edeabcf1e1634ef632b0fbc9a75b46e974910cd82"
Jul 08 17:31:56.061050 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:56.060930    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ea31b40e3b7f07a081dd6edeabcf1e1634ef632b0fbc9a75b46e974910cd82"} err="failed to get container status \"63ea31b40e3b7f07a081dd6edeabcf1e1634ef632b0fbc9a75b46e974910cd82\": rpc error: code = NotFound desc = could not find container \"63ea31b40e3b7f07a081dd6edeabcf1e1634ef632b0fbc9a75b46e974910cd82\": container with ID starting with 63ea31b40e3b7f07a081dd6edeabcf1e1634ef632b0fbc9a75b46e974910cd82 not found: ID does not exist"
Jul 08 17:31:56.064357 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:56.064338    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj"]
Jul 08 17:31:56.071617 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:56.071597    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/prestop-hook-test-kserve-router-scheduler-767d7586b8-gptkj"]
Jul 08 17:31:57.821470 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:57.821435    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="072e12d8-adce-4dc3-973a-eb218dd457d1" path="/var/lib/kubelet/pods/072e12d8-adce-4dc3-973a-eb218dd457d1/volumes"
Jul 08 17:31:59.202353 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:31:59.202318    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:32:08.947418 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:08.947388    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl"
Jul 08 17:32:10.585864 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:10.585823    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:32:10.604791 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:10.604727    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl"]
Jul 08 17:32:10.614070 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:32:10.614038    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 17:32:10.614070 ip-10-0-128-142 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:32:10.614070 ip-10-0-128-142 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 08 17:32:10.614070 ip-10-0-128-142 kubenswrapper[2565]:  > containerID="3532dc48786e5c8a0c67e415751a1bc017ac39f330a5402efc5b3e31421d00ad" cmd=["/bin/sleep","15"]
Jul 08 17:32:10.614372 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:10.614084    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl" message="" err=<
Jul 08 17:32:10.614372 ip-10-0-128-142 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:32:10.614372 ip-10-0-128-142 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 08 17:32:10.614372 ip-10-0-128-142 kubenswrapper[2565]:  >
Jul 08 17:32:10.614372 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:32:10.614122    2565 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 17:32:10.614372 ip-10-0-128-142 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:32:10.614372 ip-10-0-128-142 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 08 17:32:10.614372 ip-10-0-128-142 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl" podUID="8b659b11-9b1c-4d88-8e76-5144b0c43758" containerName="main" containerID="cri-o://3532dc48786e5c8a0c67e415751a1bc017ac39f330a5402efc5b3e31421d00ad"
Jul 08 17:32:10.614372 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:10.614181    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl" podUID="8b659b11-9b1c-4d88-8e76-5144b0c43758" containerName="main" containerID="cri-o://3532dc48786e5c8a0c67e415751a1bc017ac39f330a5402efc5b3e31421d00ad" gracePeriod=60
Jul 08 17:32:10.680857 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:10.680825    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:32:10.863348 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:10.863322    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl"
Jul 08 17:32:10.982926 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:10.982885    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8b659b11-9b1c-4d88-8e76-5144b0c43758-tokenizer-uds\") pod \"8b659b11-9b1c-4d88-8e76-5144b0c43758\" (UID: \"8b659b11-9b1c-4d88-8e76-5144b0c43758\") "
Jul 08 17:32:10.983121 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:10.982976    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zwmx6\" (UniqueName: \"kubernetes.io/projected/8b659b11-9b1c-4d88-8e76-5144b0c43758-kube-api-access-zwmx6\") pod \"8b659b11-9b1c-4d88-8e76-5144b0c43758\" (UID: \"8b659b11-9b1c-4d88-8e76-5144b0c43758\") "
Jul 08 17:32:10.983121 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:10.983044    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b659b11-9b1c-4d88-8e76-5144b0c43758-tls-certs\") pod \"8b659b11-9b1c-4d88-8e76-5144b0c43758\" (UID: \"8b659b11-9b1c-4d88-8e76-5144b0c43758\") "
Jul 08 17:32:10.984307 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:10.983976    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b659b11-9b1c-4d88-8e76-5144b0c43758-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "8b659b11-9b1c-4d88-8e76-5144b0c43758" (UID: "8b659b11-9b1c-4d88-8e76-5144b0c43758"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:32:10.986207 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:10.986177    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b659b11-9b1c-4d88-8e76-5144b0c43758-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8b659b11-9b1c-4d88-8e76-5144b0c43758" (UID: "8b659b11-9b1c-4d88-8e76-5144b0c43758"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:32:10.986301 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:10.986255    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b659b11-9b1c-4d88-8e76-5144b0c43758-kube-api-access-zwmx6" (OuterVolumeSpecName: "kube-api-access-zwmx6") pod "8b659b11-9b1c-4d88-8e76-5144b0c43758" (UID: "8b659b11-9b1c-4d88-8e76-5144b0c43758"). InnerVolumeSpecName "kube-api-access-zwmx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:32:11.083923 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:11.083891    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b659b11-9b1c-4d88-8e76-5144b0c43758-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:32:11.083923 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:11.083917    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/8b659b11-9b1c-4d88-8e76-5144b0c43758-tokenizer-uds\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:32:11.083923 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:11.083928    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwmx6\" (UniqueName: \"kubernetes.io/projected/8b659b11-9b1c-4d88-8e76-5144b0c43758-kube-api-access-zwmx6\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:32:11.121149 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:11.121068    2565 generic.go:356] "Generic (PLEG): container finished" podID="8b659b11-9b1c-4d88-8e76-5144b0c43758" containerID="3532dc48786e5c8a0c67e415751a1bc017ac39f330a5402efc5b3e31421d00ad" exitCode=0
Jul 08 17:32:11.121149 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:11.121111    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl"
Jul 08 17:32:11.121149 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:11.121122    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl" event={"ID":"8b659b11-9b1c-4d88-8e76-5144b0c43758","Type":"ContainerDied","Data":"3532dc48786e5c8a0c67e415751a1bc017ac39f330a5402efc5b3e31421d00ad"}
Jul 08 17:32:11.121369 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:11.121168    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl" event={"ID":"8b659b11-9b1c-4d88-8e76-5144b0c43758","Type":"ContainerDied","Data":"586216eb990c7d667465f680dec5e22b8933e2b494bf4a0e6349808cca935f68"}
Jul 08 17:32:11.121369 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:11.121190    2565 scope.go:117] "RemoveContainer" containerID="3532dc48786e5c8a0c67e415751a1bc017ac39f330a5402efc5b3e31421d00ad"
Jul 08 17:32:11.131232 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:11.131213    2565 scope.go:117] "RemoveContainer" containerID="3532dc48786e5c8a0c67e415751a1bc017ac39f330a5402efc5b3e31421d00ad"
Jul 08 17:32:11.131482 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:32:11.131462    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3532dc48786e5c8a0c67e415751a1bc017ac39f330a5402efc5b3e31421d00ad\": container with ID starting with 3532dc48786e5c8a0c67e415751a1bc017ac39f330a5402efc5b3e31421d00ad not found: ID does not exist" containerID="3532dc48786e5c8a0c67e415751a1bc017ac39f330a5402efc5b3e31421d00ad"
Jul 08 17:32:11.131532 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:11.131489    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3532dc48786e5c8a0c67e415751a1bc017ac39f330a5402efc5b3e31421d00ad"} err="failed to get container status \"3532dc48786e5c8a0c67e415751a1bc017ac39f330a5402efc5b3e31421d00ad\": rpc error: code = NotFound desc = could not find container \"3532dc48786e5c8a0c67e415751a1bc017ac39f330a5402efc5b3e31421d00ad\": container with ID starting with 3532dc48786e5c8a0c67e415751a1bc017ac39f330a5402efc5b3e31421d00ad not found: ID does not exist"
Jul 08 17:32:11.148364 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:11.148329    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl"]
Jul 08 17:32:11.151091 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:11.151069    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/llmisvc431d7e30f13554296364403a5b6365f5-kserve-router-schezdkxl"]
Jul 08 17:32:11.823774 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:11.823734    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b659b11-9b1c-4d88-8e76-5144b0c43758" path="/var/lib/kubelet/pods/8b659b11-9b1c-4d88-8e76-5144b0c43758/volumes"
Jul 08 17:32:17.078120 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:17.078080    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:32:17.537443 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:17.537409    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g"]
Jul 08 17:32:17.537849 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:17.537831    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b659b11-9b1c-4d88-8e76-5144b0c43758" containerName="main"
Jul 08 17:32:17.537849 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:17.537850    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b659b11-9b1c-4d88-8e76-5144b0c43758" containerName="main"
Jul 08 17:32:17.537987 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:17.537862    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="072e12d8-adce-4dc3-973a-eb218dd457d1" containerName="main"
Jul 08 17:32:17.537987 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:17.537868    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="072e12d8-adce-4dc3-973a-eb218dd457d1" containerName="main"
Jul 08 17:32:17.537987 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:17.537876    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="072e12d8-adce-4dc3-973a-eb218dd457d1" containerName="storage-initializer"
Jul 08 17:32:17.537987 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:17.537884    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="072e12d8-adce-4dc3-973a-eb218dd457d1" containerName="storage-initializer"
Jul 08 17:32:17.537987 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:17.537952    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b659b11-9b1c-4d88-8e76-5144b0c43758" containerName="main"
Jul 08 17:32:17.537987 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:17.537964    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="072e12d8-adce-4dc3-973a-eb218dd457d1" containerName="main"
Jul 08 17:32:17.542307 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:17.542280    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g"
Jul 08 17:32:17.550056 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:17.550033    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"rolling-upgrade-test-epp-sa-dockercfg-w44zr\""
Jul 08 17:32:17.551153 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:17.551133    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qxdr\""
Jul 08 17:32:17.551233 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:17.551133    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"rolling-upgrade-test-kserve-self-signed-certs\""
Jul 08 17:32:17.580131 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:17.580099    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g"]
Jul 08 17:32:17.582286 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:17.582259    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:32:17.641092 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:17.641058    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e215f099-e763-41d2-a802-19c3f22915df-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g\" (UID: \"e215f099-e763-41d2-a802-19c3f22915df\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g"
Jul 08 17:32:17.641268 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:17.641129    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e215f099-e763-41d2-a802-19c3f22915df-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g\" (UID: \"e215f099-e763-41d2-a802-19c3f22915df\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g"
Jul 08 17:32:17.641268 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:17.641157    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzkpc\" (UniqueName: \"kubernetes.io/projected/e215f099-e763-41d2-a802-19c3f22915df-kube-api-access-tzkpc\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g\" (UID: \"e215f099-e763-41d2-a802-19c3f22915df\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g"
Jul 08 17:32:17.742455 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:17.742388    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e215f099-e763-41d2-a802-19c3f22915df-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g\" (UID: \"e215f099-e763-41d2-a802-19c3f22915df\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g"
Jul 08 17:32:17.742638 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:17.742596    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e215f099-e763-41d2-a802-19c3f22915df-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g\" (UID: \"e215f099-e763-41d2-a802-19c3f22915df\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g"
Jul 08 17:32:17.742721 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:17.742639    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzkpc\" (UniqueName: \"kubernetes.io/projected/e215f099-e763-41d2-a802-19c3f22915df-kube-api-access-tzkpc\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g\" (UID: \"e215f099-e763-41d2-a802-19c3f22915df\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g"
Jul 08 17:32:17.743455 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:17.743430    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e215f099-e763-41d2-a802-19c3f22915df-tokenizer-uds\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g\" (UID: \"e215f099-e763-41d2-a802-19c3f22915df\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g"
Jul 08 17:32:17.745427 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:17.745406    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e215f099-e763-41d2-a802-19c3f22915df-tls-certs\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g\" (UID: \"e215f099-e763-41d2-a802-19c3f22915df\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g"
Jul 08 17:32:17.760632 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:17.760611    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzkpc\" (UniqueName: \"kubernetes.io/projected/e215f099-e763-41d2-a802-19c3f22915df-kube-api-access-tzkpc\") pod \"rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g\" (UID: \"e215f099-e763-41d2-a802-19c3f22915df\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g"
Jul 08 17:32:17.851195 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:17.851119    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g"
Jul 08 17:32:18.006527 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:18.006473    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g"]
Jul 08 17:32:18.009185 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:32:18.009157    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode215f099_e763_41d2_a802_19c3f22915df.slice/crio-c01f1bea3d89fdf1b1b23aa8ad00c1f00313b576971ee3d12c614a1d558157bd WatchSource:0}: Error finding container c01f1bea3d89fdf1b1b23aa8ad00c1f00313b576971ee3d12c614a1d558157bd: Status 404 returned error can't find the container with id c01f1bea3d89fdf1b1b23aa8ad00c1f00313b576971ee3d12c614a1d558157bd
Jul 08 17:32:18.164068 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:18.163968    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g" event={"ID":"e215f099-e763-41d2-a802-19c3f22915df","Type":"ContainerStarted","Data":"2b4a482c18b7b6e9cf017c5f85944635a3921cfb645cb227cbb0a5bc7afe8ab0"}
Jul 08 17:32:18.164068 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:18.164037    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g"
Jul 08 17:32:18.164068 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:18.164058    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g" event={"ID":"e215f099-e763-41d2-a802-19c3f22915df","Type":"ContainerStarted","Data":"c01f1bea3d89fdf1b1b23aa8ad00c1f00313b576971ee3d12c614a1d558157bd"}
Jul 08 17:32:18.228769 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:18.228709    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g" podStartSLOduration=1.228664187 podStartE2EDuration="1.228664187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:32:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:32:18.227455849 +0000 UTC m=+3474.940817285" watchObservedRunningTime="2026-07-08 17:32:18.228664187 +0000 UTC m=+3474.942025612"
Jul 08 17:32:31.780666 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:31.780627    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:32:38.681790 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:38.681754    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:32:39.135447 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:39.135352    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:32:49.172298 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:49.172219    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g"
Jul 08 17:32:49.207438 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:49.207393    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:32:52.245511 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:52.245470    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"]
Jul 08 17:32:52.249436 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:52.249409    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"
Jul 08 17:32:52.279035 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:52.279001    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"]
Jul 08 17:32:52.363297 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:52.363255    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-tmp-dir\") pod \"rolling-upgrade-test-kserve-859df57bcb-7zscs\" (UID: \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"
Jul 08 17:32:52.363477 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:52.363357    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-model-cache\") pod \"rolling-upgrade-test-kserve-859df57bcb-7zscs\" (UID: \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"
Jul 08 17:32:52.363477 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:52.363403    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-dshm\") pod \"rolling-upgrade-test-kserve-859df57bcb-7zscs\" (UID: \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"
Jul 08 17:32:52.363477 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:52.363423    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-tls-certs\") pod \"rolling-upgrade-test-kserve-859df57bcb-7zscs\" (UID: \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"
Jul 08 17:32:52.363477 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:52.363437    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-home\") pod \"rolling-upgrade-test-kserve-859df57bcb-7zscs\" (UID: \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"
Jul 08 17:32:52.363477 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:52.363473    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcnpg\" (UniqueName: \"kubernetes.io/projected/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-kube-api-access-mcnpg\") pod \"rolling-upgrade-test-kserve-859df57bcb-7zscs\" (UID: \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"
Jul 08 17:32:52.464513 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:52.464478    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-model-cache\") pod \"rolling-upgrade-test-kserve-859df57bcb-7zscs\" (UID: \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"
Jul 08 17:32:52.464698 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:52.464533    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-dshm\") pod \"rolling-upgrade-test-kserve-859df57bcb-7zscs\" (UID: \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"
Jul 08 17:32:52.464698 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:52.464562    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-tls-certs\") pod \"rolling-upgrade-test-kserve-859df57bcb-7zscs\" (UID: \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"
Jul 08 17:32:52.464698 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:52.464584    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-home\") pod \"rolling-upgrade-test-kserve-859df57bcb-7zscs\" (UID: \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"
Jul 08 17:32:52.464698 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:52.464627    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcnpg\" (UniqueName: \"kubernetes.io/projected/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-kube-api-access-mcnpg\") pod \"rolling-upgrade-test-kserve-859df57bcb-7zscs\" (UID: \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"
Jul 08 17:32:52.464883 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:52.464721    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-tmp-dir\") pod \"rolling-upgrade-test-kserve-859df57bcb-7zscs\" (UID: \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"
Jul 08 17:32:52.464973 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:52.464954    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-model-cache\") pod \"rolling-upgrade-test-kserve-859df57bcb-7zscs\" (UID: \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"
Jul 08 17:32:52.465043 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:52.465022    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-home\") pod \"rolling-upgrade-test-kserve-859df57bcb-7zscs\" (UID: \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"
Jul 08 17:32:52.465098 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:52.465087    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-tmp-dir\") pod \"rolling-upgrade-test-kserve-859df57bcb-7zscs\" (UID: \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"
Jul 08 17:32:52.466888 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:52.466869    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-dshm\") pod \"rolling-upgrade-test-kserve-859df57bcb-7zscs\" (UID: \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"
Jul 08 17:32:52.467064 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:52.467047    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-tls-certs\") pod \"rolling-upgrade-test-kserve-859df57bcb-7zscs\" (UID: \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"
Jul 08 17:32:52.486500 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:52.486476    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcnpg\" (UniqueName: \"kubernetes.io/projected/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-kube-api-access-mcnpg\") pod \"rolling-upgrade-test-kserve-859df57bcb-7zscs\" (UID: \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\") " pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"
Jul 08 17:32:52.560455 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:52.560364    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"
Jul 08 17:32:52.710733 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:52.710707    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"]
Jul 08 17:32:52.712895 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:32:52.712871    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe3482b4_a9dd_442d_8021_1eeb38ee7e7c.slice/crio-e4bd17299f7756a0d00f99e5f409ae4f4f0bfa4e5c71a3ef2e63f41c1bf90e2e WatchSource:0}: Error finding container e4bd17299f7756a0d00f99e5f409ae4f4f0bfa4e5c71a3ef2e63f41c1bf90e2e: Status 404 returned error can't find the container with id e4bd17299f7756a0d00f99e5f409ae4f4f0bfa4e5c71a3ef2e63f41c1bf90e2e
Jul 08 17:32:53.360395 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:53.360362    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs" event={"ID":"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c","Type":"ContainerStarted","Data":"cd78ef78a55699f9152653d6a43408c7e9e5bb5575f28550984ebdb0206e7652"}
Jul 08 17:32:53.360795 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:53.360418    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs" event={"ID":"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c","Type":"ContainerStarted","Data":"e4bd17299f7756a0d00f99e5f409ae4f4f0bfa4e5c71a3ef2e63f41c1bf90e2e"}
Jul 08 17:32:53.387363 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:53.387314    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs" podStartSLOduration=1.387299528 podStartE2EDuration="1.387299528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:32:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:32:53.385078592 +0000 UTC m=+3510.098440038" watchObservedRunningTime="2026-07-08 17:32:53.387299528 +0000 UTC m=+3510.100660967"
Jul 08 17:32:53.561113 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:32:53.561073    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"
Jul 08 17:33:02.561168 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:02.561127    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"
Jul 08 17:33:02.573018 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:02.572989    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"
Jul 08 17:33:03.432449 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:03.432419    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"
Jul 08 17:33:03.589215 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:03.589157    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:33:03.612507 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:03.612475    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"]
Jul 08 17:33:03.616740 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:03.616714    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g"]
Jul 08 17:33:03.625308 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:33:03.625277    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 17:33:03.625308 ip-10-0-128-142 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:33:03.625308 ip-10-0-128-142 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 08 17:33:03.625308 ip-10-0-128-142 kubenswrapper[2565]:  > containerID="2b4a482c18b7b6e9cf017c5f85944635a3921cfb645cb227cbb0a5bc7afe8ab0" cmd=["/bin/sleep","15"]
Jul 08 17:33:03.625523 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:03.625324    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g" message="" err=<
Jul 08 17:33:03.625523 ip-10-0-128-142 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:33:03.625523 ip-10-0-128-142 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 08 17:33:03.625523 ip-10-0-128-142 kubenswrapper[2565]:  >
Jul 08 17:33:03.625523 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:33:03.625336    2565 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 17:33:03.625523 ip-10-0-128-142 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:33:03.625523 ip-10-0-128-142 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 08 17:33:03.625523 ip-10-0-128-142 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g" podUID="e215f099-e763-41d2-a802-19c3f22915df" containerName="main" containerID="cri-o://2b4a482c18b7b6e9cf017c5f85944635a3921cfb645cb227cbb0a5bc7afe8ab0"
Jul 08 17:33:03.625523 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:03.625381    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g" podUID="e215f099-e763-41d2-a802-19c3f22915df" containerName="main" containerID="cri-o://2b4a482c18b7b6e9cf017c5f85944635a3921cfb645cb227cbb0a5bc7afe8ab0" gracePeriod=60
Jul 08 17:33:03.705171 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:03.705064    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:33:03.881848 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:03.881826    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g"
Jul 08 17:33:03.965686 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:03.965578    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzkpc\" (UniqueName: \"kubernetes.io/projected/e215f099-e763-41d2-a802-19c3f22915df-kube-api-access-tzkpc\") pod \"e215f099-e763-41d2-a802-19c3f22915df\" (UID: \"e215f099-e763-41d2-a802-19c3f22915df\") "
Jul 08 17:33:03.965686 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:03.965614    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e215f099-e763-41d2-a802-19c3f22915df-tls-certs\") pod \"e215f099-e763-41d2-a802-19c3f22915df\" (UID: \"e215f099-e763-41d2-a802-19c3f22915df\") "
Jul 08 17:33:03.965931 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:03.965691    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e215f099-e763-41d2-a802-19c3f22915df-tokenizer-uds\") pod \"e215f099-e763-41d2-a802-19c3f22915df\" (UID: \"e215f099-e763-41d2-a802-19c3f22915df\") "
Jul 08 17:33:03.966034 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:03.966001    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e215f099-e763-41d2-a802-19c3f22915df-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "e215f099-e763-41d2-a802-19c3f22915df" (UID: "e215f099-e763-41d2-a802-19c3f22915df"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:33:03.967760 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:03.967737    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e215f099-e763-41d2-a802-19c3f22915df-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e215f099-e763-41d2-a802-19c3f22915df" (UID: "e215f099-e763-41d2-a802-19c3f22915df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:33:03.967851 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:03.967778    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e215f099-e763-41d2-a802-19c3f22915df-kube-api-access-tzkpc" (OuterVolumeSpecName: "kube-api-access-tzkpc") pod "e215f099-e763-41d2-a802-19c3f22915df" (UID: "e215f099-e763-41d2-a802-19c3f22915df"). InnerVolumeSpecName "kube-api-access-tzkpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:33:04.066470 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:04.066438    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/e215f099-e763-41d2-a802-19c3f22915df-tokenizer-uds\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:33:04.066470 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:04.066467    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzkpc\" (UniqueName: \"kubernetes.io/projected/e215f099-e763-41d2-a802-19c3f22915df-kube-api-access-tzkpc\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:33:04.066470 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:04.066477    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e215f099-e763-41d2-a802-19c3f22915df-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:33:04.428410 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:04.428376    2565 generic.go:356] "Generic (PLEG): container finished" podID="e215f099-e763-41d2-a802-19c3f22915df" containerID="2b4a482c18b7b6e9cf017c5f85944635a3921cfb645cb227cbb0a5bc7afe8ab0" exitCode=0
Jul 08 17:33:04.428611 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:04.428422    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g"
Jul 08 17:33:04.428611 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:04.428458    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g" event={"ID":"e215f099-e763-41d2-a802-19c3f22915df","Type":"ContainerDied","Data":"2b4a482c18b7b6e9cf017c5f85944635a3921cfb645cb227cbb0a5bc7afe8ab0"}
Jul 08 17:33:04.428611 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:04.428509    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g" event={"ID":"e215f099-e763-41d2-a802-19c3f22915df","Type":"ContainerDied","Data":"c01f1bea3d89fdf1b1b23aa8ad00c1f00313b576971ee3d12c614a1d558157bd"}
Jul 08 17:33:04.428611 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:04.428533    2565 scope.go:117] "RemoveContainer" containerID="2b4a482c18b7b6e9cf017c5f85944635a3921cfb645cb227cbb0a5bc7afe8ab0"
Jul 08 17:33:04.438449 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:04.438427    2565 scope.go:117] "RemoveContainer" containerID="2b4a482c18b7b6e9cf017c5f85944635a3921cfb645cb227cbb0a5bc7afe8ab0"
Jul 08 17:33:04.438725 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:33:04.438705    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b4a482c18b7b6e9cf017c5f85944635a3921cfb645cb227cbb0a5bc7afe8ab0\": container with ID starting with 2b4a482c18b7b6e9cf017c5f85944635a3921cfb645cb227cbb0a5bc7afe8ab0 not found: ID does not exist" containerID="2b4a482c18b7b6e9cf017c5f85944635a3921cfb645cb227cbb0a5bc7afe8ab0"
Jul 08 17:33:04.438769 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:04.438733    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b4a482c18b7b6e9cf017c5f85944635a3921cfb645cb227cbb0a5bc7afe8ab0"} err="failed to get container status \"2b4a482c18b7b6e9cf017c5f85944635a3921cfb645cb227cbb0a5bc7afe8ab0\": rpc error: code = NotFound desc = could not find container \"2b4a482c18b7b6e9cf017c5f85944635a3921cfb645cb227cbb0a5bc7afe8ab0\": container with ID starting with 2b4a482c18b7b6e9cf017c5f85944635a3921cfb645cb227cbb0a5bc7afe8ab0 not found: ID does not exist"
Jul 08 17:33:04.452389 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:04.452365    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g"]
Jul 08 17:33:04.461950 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:04.461929    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-router-scheduler-8586d4bbcd-hwz6g"]
Jul 08 17:33:04.468992 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:33:04.468971    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 08 17:33:04.469078 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:33:04.469041    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-tls-certs podName:fe3482b4-a9dd-442d-8021-1eeb38ee7e7c nodeName:}" failed. No retries permitted until 2026-07-08 17:33:04.969020852 +0000 UTC m=+3521.682382270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-tls-certs") pod "rolling-upgrade-test-kserve-859df57bcb-7zscs" (UID: "fe3482b4-a9dd-442d-8021-1eeb38ee7e7c") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 08 17:33:04.973229 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:33:04.973196    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 08 17:33:04.973670 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:33:04.973264    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-tls-certs podName:fe3482b4-a9dd-442d-8021-1eeb38ee7e7c nodeName:}" failed. No retries permitted until 2026-07-08 17:33:05.973250178 +0000 UTC m=+3522.686611583 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-tls-certs") pod "rolling-upgrade-test-kserve-859df57bcb-7zscs" (UID: "fe3482b4-a9dd-442d-8021-1eeb38ee7e7c") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 08 17:33:05.821503 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:05.821469    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e215f099-e763-41d2-a802-19c3f22915df" path="/var/lib/kubelet/pods/e215f099-e763-41d2-a802-19c3f22915df/volumes"
Jul 08 17:33:05.980365 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:33:05.980332    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 08 17:33:05.980721 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:33:05.980401    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-tls-certs podName:fe3482b4-a9dd-442d-8021-1eeb38ee7e7c nodeName:}" failed. No retries permitted until 2026-07-08 17:33:07.980388283 +0000 UTC m=+3524.693749688 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-tls-certs") pod "rolling-upgrade-test-kserve-859df57bcb-7zscs" (UID: "fe3482b4-a9dd-442d-8021-1eeb38ee7e7c") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 08 17:33:07.995238 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:33:07.995207    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 08 17:33:07.995583 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:33:07.995269    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-tls-certs podName:fe3482b4-a9dd-442d-8021-1eeb38ee7e7c nodeName:}" failed. No retries permitted until 2026-07-08 17:33:11.995252857 +0000 UTC m=+3528.708614261 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-tls-certs") pod "rolling-upgrade-test-kserve-859df57bcb-7zscs" (UID: "fe3482b4-a9dd-442d-8021-1eeb38ee7e7c") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 08 17:33:12.029605 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:33:12.029573    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 08 17:33:12.029990 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:33:12.029642    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-tls-certs podName:fe3482b4-a9dd-442d-8021-1eeb38ee7e7c nodeName:}" failed. No retries permitted until 2026-07-08 17:33:20.029627212 +0000 UTC m=+3536.742988616 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-tls-certs") pod "rolling-upgrade-test-kserve-859df57bcb-7zscs" (UID: "fe3482b4-a9dd-442d-8021-1eeb38ee7e7c") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 08 17:33:12.484964 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:12.484928    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:33:12.578580 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:12.578543    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:33:15.736573 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:15.736533    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:33:15.817743 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:15.817691    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:33:20.099691 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:33:20.099646    2565 secret.go:189] Couldn't get secret kserve-ci-e2e-test/rolling-upgrade-test-kserve-self-signed-certs: secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 08 17:33:20.100069 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:33:20.099745    2565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-tls-certs podName:fe3482b4-a9dd-442d-8021-1eeb38ee7e7c nodeName:}" failed. No retries permitted until 2026-07-08 17:33:36.099731571 +0000 UTC m=+3552.813092976 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-tls-certs") pod "rolling-upgrade-test-kserve-859df57bcb-7zscs" (UID: "fe3482b4-a9dd-442d-8021-1eeb38ee7e7c") : secret "rolling-upgrade-test-kserve-self-signed-certs" not found
Jul 08 17:33:20.454359 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:20.454301    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs" podUID="fe3482b4-a9dd-442d-8021-1eeb38ee7e7c" containerName="main" containerID="cri-o://cd78ef78a55699f9152653d6a43408c7e9e5bb5575f28550984ebdb0206e7652" gracePeriod=45
Jul 08 17:33:20.707667 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:20.707613    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"
Jul 08 17:33:20.805671 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:20.805648    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-dshm\") pod \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\" (UID: \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\") "
Jul 08 17:33:20.805806 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:20.805720    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcnpg\" (UniqueName: \"kubernetes.io/projected/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-kube-api-access-mcnpg\") pod \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\" (UID: \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\") "
Jul 08 17:33:20.805903 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:20.805882    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-tmp-dir\") pod \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\" (UID: \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\") "
Jul 08 17:33:20.805958 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:20.805947    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-tls-certs\") pod \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\" (UID: \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\") "
Jul 08 17:33:20.806000 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:20.805976    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-home\") pod \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\" (UID: \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\") "
Jul 08 17:33:20.806062 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:20.806039    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-model-cache\") pod \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\" (UID: \"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c\") "
Jul 08 17:33:20.806188 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:20.806165    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fe3482b4-a9dd-442d-8021-1eeb38ee7e7c" (UID: "fe3482b4-a9dd-442d-8021-1eeb38ee7e7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:33:20.806294 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:20.806275    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-home" (OuterVolumeSpecName: "home") pod "fe3482b4-a9dd-442d-8021-1eeb38ee7e7c" (UID: "fe3482b4-a9dd-442d-8021-1eeb38ee7e7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:33:20.806389 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:20.806366    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-model-cache" (OuterVolumeSpecName: "model-cache") pod "fe3482b4-a9dd-442d-8021-1eeb38ee7e7c" (UID: "fe3482b4-a9dd-442d-8021-1eeb38ee7e7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:33:20.806448 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:20.806408    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:33:20.806448 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:20.806424    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:33:20.807776 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:20.807749    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-kube-api-access-mcnpg" (OuterVolumeSpecName: "kube-api-access-mcnpg") pod "fe3482b4-a9dd-442d-8021-1eeb38ee7e7c" (UID: "fe3482b4-a9dd-442d-8021-1eeb38ee7e7c"). InnerVolumeSpecName "kube-api-access-mcnpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:33:20.807861 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:20.807750    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-dshm" (OuterVolumeSpecName: "dshm") pod "fe3482b4-a9dd-442d-8021-1eeb38ee7e7c" (UID: "fe3482b4-a9dd-442d-8021-1eeb38ee7e7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:33:20.807861 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:20.807813    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fe3482b4-a9dd-442d-8021-1eeb38ee7e7c" (UID: "fe3482b4-a9dd-442d-8021-1eeb38ee7e7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:33:20.906863 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:20.906843    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:33:20.906863 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:20.906862    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcnpg\" (UniqueName: \"kubernetes.io/projected/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-kube-api-access-mcnpg\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:33:20.907007 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:20.906873    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:33:20.907007 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:20.906882    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:33:21.520736 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:21.520705    2565 generic.go:356] "Generic (PLEG): container finished" podID="fe3482b4-a9dd-442d-8021-1eeb38ee7e7c" containerID="cd78ef78a55699f9152653d6a43408c7e9e5bb5575f28550984ebdb0206e7652" exitCode=0
Jul 08 17:33:21.521123 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:21.520743    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs" event={"ID":"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c","Type":"ContainerDied","Data":"cd78ef78a55699f9152653d6a43408c7e9e5bb5575f28550984ebdb0206e7652"}
Jul 08 17:33:21.521123 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:21.520783    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs" event={"ID":"fe3482b4-a9dd-442d-8021-1eeb38ee7e7c","Type":"ContainerDied","Data":"e4bd17299f7756a0d00f99e5f409ae4f4f0bfa4e5c71a3ef2e63f41c1bf90e2e"}
Jul 08 17:33:21.521123 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:21.520788    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"
Jul 08 17:33:21.521123 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:21.520803    2565 scope.go:117] "RemoveContainer" containerID="cd78ef78a55699f9152653d6a43408c7e9e5bb5575f28550984ebdb0206e7652"
Jul 08 17:33:21.529723 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:21.529702    2565 scope.go:117] "RemoveContainer" containerID="cd78ef78a55699f9152653d6a43408c7e9e5bb5575f28550984ebdb0206e7652"
Jul 08 17:33:21.529968 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:33:21.529949    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd78ef78a55699f9152653d6a43408c7e9e5bb5575f28550984ebdb0206e7652\": container with ID starting with cd78ef78a55699f9152653d6a43408c7e9e5bb5575f28550984ebdb0206e7652 not found: ID does not exist" containerID="cd78ef78a55699f9152653d6a43408c7e9e5bb5575f28550984ebdb0206e7652"
Jul 08 17:33:21.530040 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:21.529976    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd78ef78a55699f9152653d6a43408c7e9e5bb5575f28550984ebdb0206e7652"} err="failed to get container status \"cd78ef78a55699f9152653d6a43408c7e9e5bb5575f28550984ebdb0206e7652\": rpc error: code = NotFound desc = could not find container \"cd78ef78a55699f9152653d6a43408c7e9e5bb5575f28550984ebdb0206e7652\": container with ID starting with cd78ef78a55699f9152653d6a43408c7e9e5bb5575f28550984ebdb0206e7652 not found: ID does not exist"
Jul 08 17:33:21.552213 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:21.552192    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"]
Jul 08 17:33:21.570440 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:21.570414    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/rolling-upgrade-test-kserve-859df57bcb-7zscs"]
Jul 08 17:33:21.821970 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:21.821899    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe3482b4-a9dd-442d-8021-1eeb38ee7e7c" path="/var/lib/kubelet/pods/fe3482b4-a9dd-442d-8021-1eeb38ee7e7c/volumes"
Jul 08 17:33:48.665268 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:48.665233    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:33:49.024158 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:49.024113    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95"]
Jul 08 17:33:49.024628 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:49.024606    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe3482b4-a9dd-442d-8021-1eeb38ee7e7c" containerName="main"
Jul 08 17:33:49.024628 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:49.024630    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe3482b4-a9dd-442d-8021-1eeb38ee7e7c" containerName="main"
Jul 08 17:33:49.024808 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:49.024652    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e215f099-e763-41d2-a802-19c3f22915df" containerName="main"
Jul 08 17:33:49.024808 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:49.024660    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="e215f099-e763-41d2-a802-19c3f22915df" containerName="main"
Jul 08 17:33:49.024808 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:49.024779    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe3482b4-a9dd-442d-8021-1eeb38ee7e7c" containerName="main"
Jul 08 17:33:49.024808 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:49.024788    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="e215f099-e763-41d2-a802-19c3f22915df" containerName="main"
Jul 08 17:33:49.027826 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:49.027808    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95"
Jul 08 17:33:49.030820 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:49.030793    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-kserve-self-signed-certs\""
Jul 08 17:33:49.031573 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:49.031557    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-ha-replicas-test-epp-sa-dockercfg-wrhzr\""
Jul 08 17:33:49.031626 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:49.031592    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qxdr\""
Jul 08 17:33:49.042207 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:49.042183    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95"]
Jul 08 17:33:49.084316 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:49.084281    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:33:49.139655 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:49.139624    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/29362ada-c6a5-433b-bc8d-e8fdc6df2047-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95\" (UID: \"29362ada-c6a5-433b-bc8d-e8fdc6df2047\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95"
Jul 08 17:33:49.139842 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:49.139710    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29362ada-c6a5-433b-bc8d-e8fdc6df2047-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95\" (UID: \"29362ada-c6a5-433b-bc8d-e8fdc6df2047\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95"
Jul 08 17:33:49.139842 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:49.139782    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp977\" (UniqueName: \"kubernetes.io/projected/29362ada-c6a5-433b-bc8d-e8fdc6df2047-kube-api-access-bp977\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95\" (UID: \"29362ada-c6a5-433b-bc8d-e8fdc6df2047\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95"
Jul 08 17:33:49.241068 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:49.241029    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/29362ada-c6a5-433b-bc8d-e8fdc6df2047-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95\" (UID: \"29362ada-c6a5-433b-bc8d-e8fdc6df2047\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95"
Jul 08 17:33:49.241266 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:49.241096    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29362ada-c6a5-433b-bc8d-e8fdc6df2047-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95\" (UID: \"29362ada-c6a5-433b-bc8d-e8fdc6df2047\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95"
Jul 08 17:33:49.241266 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:49.241150    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp977\" (UniqueName: \"kubernetes.io/projected/29362ada-c6a5-433b-bc8d-e8fdc6df2047-kube-api-access-bp977\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95\" (UID: \"29362ada-c6a5-433b-bc8d-e8fdc6df2047\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95"
Jul 08 17:33:49.241450 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:49.241424    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/29362ada-c6a5-433b-bc8d-e8fdc6df2047-tokenizer-uds\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95\" (UID: \"29362ada-c6a5-433b-bc8d-e8fdc6df2047\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95"
Jul 08 17:33:49.243439 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:49.243410    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29362ada-c6a5-433b-bc8d-e8fdc6df2047-tls-certs\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95\" (UID: \"29362ada-c6a5-433b-bc8d-e8fdc6df2047\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95"
Jul 08 17:33:49.251122 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:49.251094    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp977\" (UniqueName: \"kubernetes.io/projected/29362ada-c6a5-433b-bc8d-e8fdc6df2047-kube-api-access-bp977\") pod \"scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95\" (UID: \"29362ada-c6a5-433b-bc8d-e8fdc6df2047\") " pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95"
Jul 08 17:33:49.337777 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:49.337651    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95"
Jul 08 17:33:49.480827 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:49.480798    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95"]
Jul 08 17:33:49.482327 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:33:49.482291    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29362ada_c6a5_433b_bc8d_e8fdc6df2047.slice/crio-10dc6720f70252c61c2c783c1ad6c9b94d4399231191d64fd3ad4e10631f3b14 WatchSource:0}: Error finding container 10dc6720f70252c61c2c783c1ad6c9b94d4399231191d64fd3ad4e10631f3b14: Status 404 returned error can't find the container with id 10dc6720f70252c61c2c783c1ad6c9b94d4399231191d64fd3ad4e10631f3b14
Jul 08 17:33:49.672647 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:49.672615    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95" event={"ID":"29362ada-c6a5-433b-bc8d-e8fdc6df2047","Type":"ContainerStarted","Data":"bf28156e2b9d044451579191275b7809aa8463888291dfb2298db72e81c4edd9"}
Jul 08 17:33:49.673043 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:49.672664    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95" event={"ID":"29362ada-c6a5-433b-bc8d-e8fdc6df2047","Type":"ContainerStarted","Data":"10dc6720f70252c61c2c783c1ad6c9b94d4399231191d64fd3ad4e10631f3b14"}
Jul 08 17:33:49.673043 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:49.672727    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95"
Jul 08 17:33:49.716742 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:49.716684    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95" podStartSLOduration=0.716655801 podStartE2EDuration="716.655801ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:33:49.714370537 +0000 UTC m=+3566.427731962" watchObservedRunningTime="2026-07-08 17:33:49.716655801 +0000 UTC m=+3566.430017226"
Jul 08 17:33:59.156363 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:33:59.156324    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:34:20.687279 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:34:20.687245    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95"
Jul 08 17:34:47.251279 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:34:47.251236    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:34:49.499699 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:34:49.499640    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:34:49.585040 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:34:49.585007    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95"]
Jul 08 17:34:49.587433 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:34:49.587399    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:34:49.593871 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:34:49.593834    2565 log.go:32] "ExecSync cmd from runtime service failed" err=<
Jul 08 17:34:49.593871 ip-10-0-128-142 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:34:49.593871 ip-10-0-128-142 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 08 17:34:49.593871 ip-10-0-128-142 kubenswrapper[2565]:  > containerID="bf28156e2b9d044451579191275b7809aa8463888291dfb2298db72e81c4edd9" cmd=["/bin/sleep","15"]
Jul 08 17:34:49.594092 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:34:49.593883    2565 handlers.go:82] "Exec lifecycle hook for Container in Pod failed" execCommand=["/bin/sleep","15"] containerName="main" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95" message="" err=<
Jul 08 17:34:49.594092 ip-10-0-128-142 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:34:49.594092 ip-10-0-128-142 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 08 17:34:49.594092 ip-10-0-128-142 kubenswrapper[2565]:  >
Jul 08 17:34:49.594092 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:34:49.593897    2565 kuberuntime_container.go:754] "PreStop hook failed" err=<
Jul 08 17:34:49.594092 ip-10-0-128-142 kubenswrapper[2565]:         rpc error: code = Unknown desc = command error: executable file `/bin/sleep` not found: No such file or directory
Jul 08 17:34:49.594092 ip-10-0-128-142 kubenswrapper[2565]:         , stdout: , stderr: , exit code -1
Jul 08 17:34:49.594092 ip-10-0-128-142 kubenswrapper[2565]:  > pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95" podUID="29362ada-c6a5-433b-bc8d-e8fdc6df2047" containerName="main" containerID="cri-o://bf28156e2b9d044451579191275b7809aa8463888291dfb2298db72e81c4edd9"
Jul 08 17:34:49.594092 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:34:49.593930    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95" podUID="29362ada-c6a5-433b-bc8d-e8fdc6df2047" containerName="main" containerID="cri-o://bf28156e2b9d044451579191275b7809aa8463888291dfb2298db72e81c4edd9" gracePeriod=60
Jul 08 17:34:49.851656 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:34:49.851633    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95"
Jul 08 17:34:49.900308 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:34:49.900281    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29362ada-c6a5-433b-bc8d-e8fdc6df2047-tls-certs\") pod \"29362ada-c6a5-433b-bc8d-e8fdc6df2047\" (UID: \"29362ada-c6a5-433b-bc8d-e8fdc6df2047\") "
Jul 08 17:34:49.900467 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:34:49.900322    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bp977\" (UniqueName: \"kubernetes.io/projected/29362ada-c6a5-433b-bc8d-e8fdc6df2047-kube-api-access-bp977\") pod \"29362ada-c6a5-433b-bc8d-e8fdc6df2047\" (UID: \"29362ada-c6a5-433b-bc8d-e8fdc6df2047\") "
Jul 08 17:34:49.900467 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:34:49.900411    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/29362ada-c6a5-433b-bc8d-e8fdc6df2047-tokenizer-uds\") pod \"29362ada-c6a5-433b-bc8d-e8fdc6df2047\" (UID: \"29362ada-c6a5-433b-bc8d-e8fdc6df2047\") "
Jul 08 17:34:49.900718 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:34:49.900686    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29362ada-c6a5-433b-bc8d-e8fdc6df2047-tokenizer-uds" (OuterVolumeSpecName: "tokenizer-uds") pod "29362ada-c6a5-433b-bc8d-e8fdc6df2047" (UID: "29362ada-c6a5-433b-bc8d-e8fdc6df2047"). InnerVolumeSpecName "tokenizer-uds". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:34:49.902335 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:34:49.902308    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29362ada-c6a5-433b-bc8d-e8fdc6df2047-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "29362ada-c6a5-433b-bc8d-e8fdc6df2047" (UID: "29362ada-c6a5-433b-bc8d-e8fdc6df2047"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:34:49.902440 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:34:49.902384    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29362ada-c6a5-433b-bc8d-e8fdc6df2047-kube-api-access-bp977" (OuterVolumeSpecName: "kube-api-access-bp977") pod "29362ada-c6a5-433b-bc8d-e8fdc6df2047" (UID: "29362ada-c6a5-433b-bc8d-e8fdc6df2047"). InnerVolumeSpecName "kube-api-access-bp977". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:34:50.001686 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:34:50.001651    2565 reconciler_common.go:299] "Volume detached for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/29362ada-c6a5-433b-bc8d-e8fdc6df2047-tokenizer-uds\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:34:50.001686 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:34:50.001691    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29362ada-c6a5-433b-bc8d-e8fdc6df2047-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:34:50.001880 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:34:50.001701    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp977\" (UniqueName: \"kubernetes.io/projected/29362ada-c6a5-433b-bc8d-e8fdc6df2047-kube-api-access-bp977\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:34:50.005930 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:34:50.005903    2565 generic.go:356] "Generic (PLEG): container finished" podID="29362ada-c6a5-433b-bc8d-e8fdc6df2047" containerID="bf28156e2b9d044451579191275b7809aa8463888291dfb2298db72e81c4edd9" exitCode=0
Jul 08 17:34:50.006052 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:34:50.005945    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95"
Jul 08 17:34:50.006052 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:34:50.005979    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95" event={"ID":"29362ada-c6a5-433b-bc8d-e8fdc6df2047","Type":"ContainerDied","Data":"bf28156e2b9d044451579191275b7809aa8463888291dfb2298db72e81c4edd9"}
Jul 08 17:34:50.006052 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:34:50.006029    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95" event={"ID":"29362ada-c6a5-433b-bc8d-e8fdc6df2047","Type":"ContainerDied","Data":"10dc6720f70252c61c2c783c1ad6c9b94d4399231191d64fd3ad4e10631f3b14"}
Jul 08 17:34:50.006052 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:34:50.006048    2565 scope.go:117] "RemoveContainer" containerID="bf28156e2b9d044451579191275b7809aa8463888291dfb2298db72e81c4edd9"
Jul 08 17:34:50.017812 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:34:50.017791    2565 scope.go:117] "RemoveContainer" containerID="bf28156e2b9d044451579191275b7809aa8463888291dfb2298db72e81c4edd9"
Jul 08 17:34:50.018120 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:34:50.018100    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf28156e2b9d044451579191275b7809aa8463888291dfb2298db72e81c4edd9\": container with ID starting with bf28156e2b9d044451579191275b7809aa8463888291dfb2298db72e81c4edd9 not found: ID does not exist" containerID="bf28156e2b9d044451579191275b7809aa8463888291dfb2298db72e81c4edd9"
Jul 08 17:34:50.018207 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:34:50.018128    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf28156e2b9d044451579191275b7809aa8463888291dfb2298db72e81c4edd9"} err="failed to get container status \"bf28156e2b9d044451579191275b7809aa8463888291dfb2298db72e81c4edd9\": rpc error: code = NotFound desc = could not find container \"bf28156e2b9d044451579191275b7809aa8463888291dfb2298db72e81c4edd9\": container with ID starting with bf28156e2b9d044451579191275b7809aa8463888291dfb2298db72e81c4edd9 not found: ID does not exist"
Jul 08 17:34:50.046283 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:34:50.046252    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95"]
Jul 08 17:34:50.052851 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:34:50.052828    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-ha-replicas-test-kserve-router-scheduler-6f5f85d9dg95"]
Jul 08 17:34:51.823049 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:34:51.823012    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29362ada-c6a5-433b-bc8d-e8fdc6df2047" path="/var/lib/kubelet/pods/29362ada-c6a5-433b-bc8d-e8fdc6df2047/volumes"
Jul 08 17:35:20.074021 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:20.073980    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:35:20.321624 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:20.321586    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74"]
Jul 08 17:35:20.322114 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:20.322093    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29362ada-c6a5-433b-bc8d-e8fdc6df2047" containerName="main"
Jul 08 17:35:20.322114 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:20.322113    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="29362ada-c6a5-433b-bc8d-e8fdc6df2047" containerName="main"
Jul 08 17:35:20.322247 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:20.322201    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="29362ada-c6a5-433b-bc8d-e8fdc6df2047" containerName="main"
Jul 08 17:35:20.325293 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:20.325249    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74"
Jul 08 17:35:20.328909 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:20.328890    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-8qxdr\""
Jul 08 17:35:20.329032 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:20.328906    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-kserve-self-signed-certs\""
Jul 08 17:35:20.329032 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:20.328890    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-pd-migration-test-epp-sa-dockercfg-zdc7b\""
Jul 08 17:35:20.336535 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:20.336513    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74"]
Jul 08 17:35:20.349293 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:20.349259    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:35:20.391871 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:20.391838    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e3eb6e-ca4d-41fd-a402-927efffc9858-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74\" (UID: \"c6e3eb6e-ca4d-41fd-a402-927efffc9858\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74"
Jul 08 17:35:20.392046 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:20.391921    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr7wx\" (UniqueName: \"kubernetes.io/projected/c6e3eb6e-ca4d-41fd-a402-927efffc9858-kube-api-access-vr7wx\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74\" (UID: \"c6e3eb6e-ca4d-41fd-a402-927efffc9858\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74"
Jul 08 17:35:20.392046 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:20.391954    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c6e3eb6e-ca4d-41fd-a402-927efffc9858-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74\" (UID: \"c6e3eb6e-ca4d-41fd-a402-927efffc9858\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74"
Jul 08 17:35:20.493329 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:20.493295    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr7wx\" (UniqueName: \"kubernetes.io/projected/c6e3eb6e-ca4d-41fd-a402-927efffc9858-kube-api-access-vr7wx\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74\" (UID: \"c6e3eb6e-ca4d-41fd-a402-927efffc9858\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74"
Jul 08 17:35:20.493512 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:20.493344    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c6e3eb6e-ca4d-41fd-a402-927efffc9858-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74\" (UID: \"c6e3eb6e-ca4d-41fd-a402-927efffc9858\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74"
Jul 08 17:35:20.493512 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:20.493386    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e3eb6e-ca4d-41fd-a402-927efffc9858-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74\" (UID: \"c6e3eb6e-ca4d-41fd-a402-927efffc9858\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74"
Jul 08 17:35:20.493789 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:20.493771    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tokenizer-uds\" (UniqueName: \"kubernetes.io/empty-dir/c6e3eb6e-ca4d-41fd-a402-927efffc9858-tokenizer-uds\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74\" (UID: \"c6e3eb6e-ca4d-41fd-a402-927efffc9858\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74"
Jul 08 17:35:20.495875 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:20.495857    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6e3eb6e-ca4d-41fd-a402-927efffc9858-tls-certs\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74\" (UID: \"c6e3eb6e-ca4d-41fd-a402-927efffc9858\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74"
Jul 08 17:35:20.503882 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:20.503861    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr7wx\" (UniqueName: \"kubernetes.io/projected/c6e3eb6e-ca4d-41fd-a402-927efffc9858-kube-api-access-vr7wx\") pod \"scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74\" (UID: \"c6e3eb6e-ca4d-41fd-a402-927efffc9858\") " pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74"
Jul 08 17:35:20.635526 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:20.635444    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74"
Jul 08 17:35:20.779072 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:20.779046    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74"]
Jul 08 17:35:20.780506 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:35:20.780479    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e3eb6e_ca4d_41fd_a402_927efffc9858.slice/crio-1a89d8b51e752e87e998be92b8be61449a90750ca4dc2e3b3ddc0240b72e7b97 WatchSource:0}: Error finding container 1a89d8b51e752e87e998be92b8be61449a90750ca4dc2e3b3ddc0240b72e7b97: Status 404 returned error can't find the container with id 1a89d8b51e752e87e998be92b8be61449a90750ca4dc2e3b3ddc0240b72e7b97
Jul 08 17:35:21.174110 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:21.173415    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74" event={"ID":"c6e3eb6e-ca4d-41fd-a402-927efffc9858","Type":"ContainerStarted","Data":"f8de174ffa63fef37d2487b3f3e38fb5583e68af440801beefea57bfc6f699bb"}
Jul 08 17:35:21.174110 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:21.173474    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74" event={"ID":"c6e3eb6e-ca4d-41fd-a402-927efffc9858","Type":"ContainerStarted","Data":"1a89d8b51e752e87e998be92b8be61449a90750ca4dc2e3b3ddc0240b72e7b97"}
Jul 08 17:35:21.174110 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:21.173725    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74"
Jul 08 17:35:21.199378 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:21.199321    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74" podStartSLOduration=1.1993056659999999 podStartE2EDuration="1.199305666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:35:21.197926214 +0000 UTC m=+3657.911287640" watchObservedRunningTime="2026-07-08 17:35:21.199305666 +0000 UTC m=+3657.912667092"
Jul 08 17:35:27.755051 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:27.755021    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"]
Jul 08 17:35:27.759971 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:27.759949    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"
Jul 08 17:35:27.762723 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:27.762705    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-custom-template-test-kserve-self-signed-certs\""
Jul 08 17:35:27.770920 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:27.770895    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"]
Jul 08 17:35:27.790456 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:27.790425    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:35:27.858437 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:27.858402    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/089c050e-01d6-40bc-87e5-b7fb1822edc9-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p\" (UID: \"089c050e-01d6-40bc-87e5-b7fb1822edc9\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"
Jul 08 17:35:27.858606 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:27.858443    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/089c050e-01d6-40bc-87e5-b7fb1822edc9-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p\" (UID: \"089c050e-01d6-40bc-87e5-b7fb1822edc9\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"
Jul 08 17:35:27.858606 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:27.858503    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089c050e-01d6-40bc-87e5-b7fb1822edc9-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p\" (UID: \"089c050e-01d6-40bc-87e5-b7fb1822edc9\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"
Jul 08 17:35:27.858606 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:27.858522    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xmd8\" (UniqueName: \"kubernetes.io/projected/089c050e-01d6-40bc-87e5-b7fb1822edc9-kube-api-access-8xmd8\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p\" (UID: \"089c050e-01d6-40bc-87e5-b7fb1822edc9\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"
Jul 08 17:35:27.858606 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:27.858539    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/089c050e-01d6-40bc-87e5-b7fb1822edc9-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p\" (UID: \"089c050e-01d6-40bc-87e5-b7fb1822edc9\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"
Jul 08 17:35:27.858771 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:27.858611    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/089c050e-01d6-40bc-87e5-b7fb1822edc9-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p\" (UID: \"089c050e-01d6-40bc-87e5-b7fb1822edc9\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"
Jul 08 17:35:27.959988 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:27.959952    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/089c050e-01d6-40bc-87e5-b7fb1822edc9-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p\" (UID: \"089c050e-01d6-40bc-87e5-b7fb1822edc9\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"
Jul 08 17:35:27.960176 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:27.960044    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/089c050e-01d6-40bc-87e5-b7fb1822edc9-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p\" (UID: \"089c050e-01d6-40bc-87e5-b7fb1822edc9\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"
Jul 08 17:35:27.960176 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:27.960076    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/089c050e-01d6-40bc-87e5-b7fb1822edc9-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p\" (UID: \"089c050e-01d6-40bc-87e5-b7fb1822edc9\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"
Jul 08 17:35:27.960176 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:27.960120    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089c050e-01d6-40bc-87e5-b7fb1822edc9-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p\" (UID: \"089c050e-01d6-40bc-87e5-b7fb1822edc9\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"
Jul 08 17:35:27.960176 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:27.960152    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xmd8\" (UniqueName: \"kubernetes.io/projected/089c050e-01d6-40bc-87e5-b7fb1822edc9-kube-api-access-8xmd8\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p\" (UID: \"089c050e-01d6-40bc-87e5-b7fb1822edc9\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"
Jul 08 17:35:27.960381 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:27.960177    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/089c050e-01d6-40bc-87e5-b7fb1822edc9-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p\" (UID: \"089c050e-01d6-40bc-87e5-b7fb1822edc9\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"
Jul 08 17:35:27.960606 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:27.960574    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/089c050e-01d6-40bc-87e5-b7fb1822edc9-model-cache\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p\" (UID: \"089c050e-01d6-40bc-87e5-b7fb1822edc9\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"
Jul 08 17:35:27.960758 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:27.960615    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089c050e-01d6-40bc-87e5-b7fb1822edc9-tmp-dir\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p\" (UID: \"089c050e-01d6-40bc-87e5-b7fb1822edc9\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"
Jul 08 17:35:27.960758 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:27.960718    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/089c050e-01d6-40bc-87e5-b7fb1822edc9-home\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p\" (UID: \"089c050e-01d6-40bc-87e5-b7fb1822edc9\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"
Jul 08 17:35:27.962696 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:27.962656    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/089c050e-01d6-40bc-87e5-b7fb1822edc9-dshm\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p\" (UID: \"089c050e-01d6-40bc-87e5-b7fb1822edc9\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"
Jul 08 17:35:27.962784 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:27.962709    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/089c050e-01d6-40bc-87e5-b7fb1822edc9-tls-certs\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p\" (UID: \"089c050e-01d6-40bc-87e5-b7fb1822edc9\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"
Jul 08 17:35:27.971163 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:27.971134    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xmd8\" (UniqueName: \"kubernetes.io/projected/089c050e-01d6-40bc-87e5-b7fb1822edc9-kube-api-access-8xmd8\") pod \"scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p\" (UID: \"089c050e-01d6-40bc-87e5-b7fb1822edc9\") " pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"
Jul 08 17:35:28.071657 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:28.071564    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"
Jul 08 17:35:28.210670 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:28.210642    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"]
Jul 08 17:35:28.212546 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:35:28.212517    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod089c050e_01d6_40bc_87e5_b7fb1822edc9.slice/crio-ae8333a6f4a680f7c8e326354dc1b07fb15eb5e546e0374f6bba9990c8c77a46 WatchSource:0}: Error finding container ae8333a6f4a680f7c8e326354dc1b07fb15eb5e546e0374f6bba9990c8c77a46: Status 404 returned error can't find the container with id ae8333a6f4a680f7c8e326354dc1b07fb15eb5e546e0374f6bba9990c8c77a46
Jul 08 17:35:28.286799 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:28.286754    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:35:29.218206 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:29.218171    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p" event={"ID":"089c050e-01d6-40bc-87e5-b7fb1822edc9","Type":"ContainerStarted","Data":"829b73e452a791eb94bfe2419c96bd232a7e3b65e2519c8b7325a9da07a00b80"}
Jul 08 17:35:29.218630 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:29.218225    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p" event={"ID":"089c050e-01d6-40bc-87e5-b7fb1822edc9","Type":"ContainerStarted","Data":"ae8333a6f4a680f7c8e326354dc1b07fb15eb5e546e0374f6bba9990c8c77a46"}
Jul 08 17:35:29.257778 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:29.257729    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p" podStartSLOduration=2.257714845 podStartE2EDuration="2.257714845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:35:29.252792001 +0000 UTC m=+3665.966153450" watchObservedRunningTime="2026-07-08 17:35:29.257714845 +0000 UTC m=+3665.971076271"
Jul 08 17:35:30.071743 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:30.071703    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"
Jul 08 17:35:34.989209 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:34.989179    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:35:38.071778 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:38.071742    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"
Jul 08 17:35:38.083681 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:38.083656    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"
Jul 08 17:35:38.277953 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:38.277848    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"
Jul 08 17:35:40.905226 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:40.905184    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:35:48.867165 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:48.867068    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:35:52.181477 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:35:52.181449    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74"
Jul 08 17:36:01.654581 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:01.654543    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"]
Jul 08 17:36:01.684112 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:01.684080    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:36:01.975033 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:01.975003    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:36:16.675259 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:16.675199    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p" podUID="089c050e-01d6-40bc-87e5-b7fb1822edc9" containerName="main" containerID="cri-o://829b73e452a791eb94bfe2419c96bd232a7e3b65e2519c8b7325a9da07a00b80" gracePeriod=45
Jul 08 17:36:16.927605 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:16.927543    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"
Jul 08 17:36:16.999972 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:16.999925    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089c050e-01d6-40bc-87e5-b7fb1822edc9-tmp-dir\") pod \"089c050e-01d6-40bc-87e5-b7fb1822edc9\" (UID: \"089c050e-01d6-40bc-87e5-b7fb1822edc9\") "
Jul 08 17:36:17.000168 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:17.000001    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/089c050e-01d6-40bc-87e5-b7fb1822edc9-model-cache\") pod \"089c050e-01d6-40bc-87e5-b7fb1822edc9\" (UID: \"089c050e-01d6-40bc-87e5-b7fb1822edc9\") "
Jul 08 17:36:17.000168 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:17.000022    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/089c050e-01d6-40bc-87e5-b7fb1822edc9-home\") pod \"089c050e-01d6-40bc-87e5-b7fb1822edc9\" (UID: \"089c050e-01d6-40bc-87e5-b7fb1822edc9\") "
Jul 08 17:36:17.000168 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:17.000047    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xmd8\" (UniqueName: \"kubernetes.io/projected/089c050e-01d6-40bc-87e5-b7fb1822edc9-kube-api-access-8xmd8\") pod \"089c050e-01d6-40bc-87e5-b7fb1822edc9\" (UID: \"089c050e-01d6-40bc-87e5-b7fb1822edc9\") "
Jul 08 17:36:17.000168 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:17.000071    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/089c050e-01d6-40bc-87e5-b7fb1822edc9-tls-certs\") pod \"089c050e-01d6-40bc-87e5-b7fb1822edc9\" (UID: \"089c050e-01d6-40bc-87e5-b7fb1822edc9\") "
Jul 08 17:36:17.000168 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:17.000090    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/089c050e-01d6-40bc-87e5-b7fb1822edc9-dshm\") pod \"089c050e-01d6-40bc-87e5-b7fb1822edc9\" (UID: \"089c050e-01d6-40bc-87e5-b7fb1822edc9\") "
Jul 08 17:36:17.000411 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:17.000242    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089c050e-01d6-40bc-87e5-b7fb1822edc9-model-cache" (OuterVolumeSpecName: "model-cache") pod "089c050e-01d6-40bc-87e5-b7fb1822edc9" (UID: "089c050e-01d6-40bc-87e5-b7fb1822edc9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:36:17.000411 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:17.000253    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089c050e-01d6-40bc-87e5-b7fb1822edc9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "089c050e-01d6-40bc-87e5-b7fb1822edc9" (UID: "089c050e-01d6-40bc-87e5-b7fb1822edc9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:36:17.000411 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:17.000350    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089c050e-01d6-40bc-87e5-b7fb1822edc9-home" (OuterVolumeSpecName: "home") pod "089c050e-01d6-40bc-87e5-b7fb1822edc9" (UID: "089c050e-01d6-40bc-87e5-b7fb1822edc9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:36:17.000505 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:17.000449    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/089c050e-01d6-40bc-87e5-b7fb1822edc9-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:36:17.000505 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:17.000468    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/089c050e-01d6-40bc-87e5-b7fb1822edc9-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:36:17.000505 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:17.000479    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/089c050e-01d6-40bc-87e5-b7fb1822edc9-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:36:17.002153 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:17.002122    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/089c050e-01d6-40bc-87e5-b7fb1822edc9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "089c050e-01d6-40bc-87e5-b7fb1822edc9" (UID: "089c050e-01d6-40bc-87e5-b7fb1822edc9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:36:17.002272 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:17.002179    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/089c050e-01d6-40bc-87e5-b7fb1822edc9-dshm" (OuterVolumeSpecName: "dshm") pod "089c050e-01d6-40bc-87e5-b7fb1822edc9" (UID: "089c050e-01d6-40bc-87e5-b7fb1822edc9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:36:17.002272 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:17.002184    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/089c050e-01d6-40bc-87e5-b7fb1822edc9-kube-api-access-8xmd8" (OuterVolumeSpecName: "kube-api-access-8xmd8") pod "089c050e-01d6-40bc-87e5-b7fb1822edc9" (UID: "089c050e-01d6-40bc-87e5-b7fb1822edc9"). InnerVolumeSpecName "kube-api-access-8xmd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:36:17.101362 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:17.101325    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xmd8\" (UniqueName: \"kubernetes.io/projected/089c050e-01d6-40bc-87e5-b7fb1822edc9-kube-api-access-8xmd8\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:36:17.101362 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:17.101355    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/089c050e-01d6-40bc-87e5-b7fb1822edc9-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:36:17.101362 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:17.101367    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/089c050e-01d6-40bc-87e5-b7fb1822edc9-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:36:17.479012 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:17.478980    2565 generic.go:356] "Generic (PLEG): container finished" podID="089c050e-01d6-40bc-87e5-b7fb1822edc9" containerID="829b73e452a791eb94bfe2419c96bd232a7e3b65e2519c8b7325a9da07a00b80" exitCode=0
Jul 08 17:36:17.479189 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:17.479026    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"
Jul 08 17:36:17.479189 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:17.479053    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p" event={"ID":"089c050e-01d6-40bc-87e5-b7fb1822edc9","Type":"ContainerDied","Data":"829b73e452a791eb94bfe2419c96bd232a7e3b65e2519c8b7325a9da07a00b80"}
Jul 08 17:36:17.479189 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:17.479128    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p" event={"ID":"089c050e-01d6-40bc-87e5-b7fb1822edc9","Type":"ContainerDied","Data":"ae8333a6f4a680f7c8e326354dc1b07fb15eb5e546e0374f6bba9990c8c77a46"}
Jul 08 17:36:17.479189 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:17.479150    2565 scope.go:117] "RemoveContainer" containerID="829b73e452a791eb94bfe2419c96bd232a7e3b65e2519c8b7325a9da07a00b80"
Jul 08 17:36:17.488454 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:17.488441    2565 scope.go:117] "RemoveContainer" containerID="829b73e452a791eb94bfe2419c96bd232a7e3b65e2519c8b7325a9da07a00b80"
Jul 08 17:36:17.488696 ip-10-0-128-142 kubenswrapper[2565]: E0708 17:36:17.488666    2565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"829b73e452a791eb94bfe2419c96bd232a7e3b65e2519c8b7325a9da07a00b80\": container with ID starting with 829b73e452a791eb94bfe2419c96bd232a7e3b65e2519c8b7325a9da07a00b80 not found: ID does not exist" containerID="829b73e452a791eb94bfe2419c96bd232a7e3b65e2519c8b7325a9da07a00b80"
Jul 08 17:36:17.488740 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:17.488707    2565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"829b73e452a791eb94bfe2419c96bd232a7e3b65e2519c8b7325a9da07a00b80"} err="failed to get container status \"829b73e452a791eb94bfe2419c96bd232a7e3b65e2519c8b7325a9da07a00b80\": rpc error: code = NotFound desc = could not find container \"829b73e452a791eb94bfe2419c96bd232a7e3b65e2519c8b7325a9da07a00b80\": container with ID starting with 829b73e452a791eb94bfe2419c96bd232a7e3b65e2519c8b7325a9da07a00b80 not found: ID does not exist"
Jul 08 17:36:17.503043 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:17.503022    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"]
Jul 08 17:36:17.508941 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:17.508923    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-custom-template-test-kserve-5bc7bd6864-jbf8p"]
Jul 08 17:36:17.822251 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:36:17.822172    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="089c050e-01d6-40bc-87e5-b7fb1822edc9" path="/var/lib/kubelet/pods/089c050e-01d6-40bc-87e5-b7fb1822edc9/volumes"
Jul 08 17:38:05.316161 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:38:05.316130    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:38:05.385221 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:38:05.385190    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:38:08.763463 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:38:08.763428    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:38:08.863488 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:38:08.863456    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:44:53.672218 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:44:53.672146    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:45:12.054691 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:45:12.054635    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:45:12.155604 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:45:12.155575    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:46:20.817532 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:46:20.817492    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:48:09.469251 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:48:09.469163    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:51:42.946375 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:51:42.946343    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:51:53.481122 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:51:53.481084    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:54:17.897720 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:17.897620    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74_c6e3eb6e-ca4d-41fd-a402-927efffc9858/main/0.log"
Jul 08 17:54:25.333850 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.333817    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"]
Jul 08 17:54:25.334322 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.334179    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="089c050e-01d6-40bc-87e5-b7fb1822edc9" containerName="main"
Jul 08 17:54:25.334322 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.334191    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="089c050e-01d6-40bc-87e5-b7fb1822edc9" containerName="main"
Jul 08 17:54:25.334322 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.334248    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="089c050e-01d6-40bc-87e5-b7fb1822edc9" containerName="main"
Jul 08 17:54:25.337259 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.337243    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"
Jul 08 17:54:25.341053 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.341032    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-kserve-dockercfg-vb8k4\""
Jul 08 17:54:25.341989 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.341966    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"scheduler-v06-threshold-migration-test-kserve-self-signed-certs\""
Jul 08 17:54:25.362612 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.362591    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"]
Jul 08 17:54:25.411715 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.411686    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:54:25.417702 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.417669    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a3f6e45-845f-4982-8ee3-6c163fbf4689-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx\" (UID: \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"
Jul 08 17:54:25.417800 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.417715    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a3f6e45-845f-4982-8ee3-6c163fbf4689-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx\" (UID: \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"
Jul 08 17:54:25.417800 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.417765    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmvfr\" (UniqueName: \"kubernetes.io/projected/6a3f6e45-845f-4982-8ee3-6c163fbf4689-kube-api-access-jmvfr\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx\" (UID: \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"
Jul 08 17:54:25.417883 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.417805    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a3f6e45-845f-4982-8ee3-6c163fbf4689-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx\" (UID: \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"
Jul 08 17:54:25.417883 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.417836    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a3f6e45-845f-4982-8ee3-6c163fbf4689-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx\" (UID: \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"
Jul 08 17:54:25.417883 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.417874    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a3f6e45-845f-4982-8ee3-6c163fbf4689-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx\" (UID: \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"
Jul 08 17:54:25.518462 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.518437    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmvfr\" (UniqueName: \"kubernetes.io/projected/6a3f6e45-845f-4982-8ee3-6c163fbf4689-kube-api-access-jmvfr\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx\" (UID: \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"
Jul 08 17:54:25.518567 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.518493    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a3f6e45-845f-4982-8ee3-6c163fbf4689-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx\" (UID: \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"
Jul 08 17:54:25.518567 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.518516    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a3f6e45-845f-4982-8ee3-6c163fbf4689-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx\" (UID: \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"
Jul 08 17:54:25.518567 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.518534    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a3f6e45-845f-4982-8ee3-6c163fbf4689-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx\" (UID: \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"
Jul 08 17:54:25.518764 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.518587    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a3f6e45-845f-4982-8ee3-6c163fbf4689-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx\" (UID: \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"
Jul 08 17:54:25.518764 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.518603    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a3f6e45-845f-4982-8ee3-6c163fbf4689-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx\" (UID: \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"
Jul 08 17:54:25.518973 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.518953    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a3f6e45-845f-4982-8ee3-6c163fbf4689-tmp-dir\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx\" (UID: \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"
Jul 08 17:54:25.519034 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.519002    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a3f6e45-845f-4982-8ee3-6c163fbf4689-model-cache\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx\" (UID: \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"
Jul 08 17:54:25.519096 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.519078    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a3f6e45-845f-4982-8ee3-6c163fbf4689-home\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx\" (UID: \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"
Jul 08 17:54:25.520705 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.520689    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a3f6e45-845f-4982-8ee3-6c163fbf4689-dshm\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx\" (UID: \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"
Jul 08 17:54:25.521018 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.520999    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a3f6e45-845f-4982-8ee3-6c163fbf4689-tls-certs\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx\" (UID: \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"
Jul 08 17:54:25.527524 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.527500    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmvfr\" (UniqueName: \"kubernetes.io/projected/6a3f6e45-845f-4982-8ee3-6c163fbf4689-kube-api-access-jmvfr\") pod \"scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx\" (UID: \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\") " pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"
Jul 08 17:54:25.648137 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.648057    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"
Jul 08 17:54:25.764092 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.764042    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:54:25.790384 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:25.790357    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"]
Jul 08 17:54:25.791156 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:54:25.791129    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a3f6e45_845f_4982_8ee3_6c163fbf4689.slice/crio-372d09e84980058b7f248b8a867bd2b84385bd9e5c5ff9bbf0d410464966b3ff WatchSource:0}: Error finding container 372d09e84980058b7f248b8a867bd2b84385bd9e5c5ff9bbf0d410464966b3ff: Status 404 returned error can't find the container with id 372d09e84980058b7f248b8a867bd2b84385bd9e5c5ff9bbf0d410464966b3ff
Jul 08 17:54:26.332226 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:26.332183    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx" event={"ID":"6a3f6e45-845f-4982-8ee3-6c163fbf4689","Type":"ContainerStarted","Data":"34323a2e55f0128b4f1b7fa0bb03f8c87be7f6550d3722cccc2c49938c862d3b"}
Jul 08 17:54:26.332401 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:26.332250    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx" event={"ID":"6a3f6e45-845f-4982-8ee3-6c163fbf4689","Type":"ContainerStarted","Data":"372d09e84980058b7f248b8a867bd2b84385bd9e5c5ff9bbf0d410464966b3ff"}
Jul 08 17:54:26.332401 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:26.332312    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"
Jul 08 17:54:27.341467 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:27.341432    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx" event={"ID":"6a3f6e45-845f-4982-8ee3-6c163fbf4689","Type":"ContainerStarted","Data":"df07589598269ba47d1d1ce8491438bd5eff34fabc4f8224bc07efede219e7c2"}
Jul 08 17:54:27.648557 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:27.648473    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"
Jul 08 17:54:35.648789 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:35.648696    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"
Jul 08 17:54:35.663026 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:35.662989    2565 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"
Jul 08 17:54:35.663304 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:35.663288    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"
Jul 08 17:54:35.685468 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:35.685419    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx" podStartSLOduration=10.685404703 podStartE2EDuration="10.685404703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:54:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:54:27.370251686 +0000 UTC m=+4804.083613114" watchObservedRunningTime="2026-07-08 17:54:35.685404703 +0000 UTC m=+4812.398766130"
Jul 08 17:54:36.398818 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:36.398789    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"
Jul 08 17:54:58.545235 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:54:58.545193    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:55:01.260432 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:01.260392    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:55:01.269210 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:01.269185    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"]
Jul 08 17:55:01.359604 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:01.359567    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:55:16.289665 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:16.289576    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx" podUID="6a3f6e45-845f-4982-8ee3-6c163fbf4689" containerName="main" containerID="cri-o://df07589598269ba47d1d1ce8491438bd5eff34fabc4f8224bc07efede219e7c2" gracePeriod=45
Jul 08 17:55:16.388976 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:16.388930    2565 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx" podUID="6a3f6e45-845f-4982-8ee3-6c163fbf4689" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8001/health\": dial tcp 10.133.0.86:8001: connect: connection refused"
Jul 08 17:55:16.407387 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:16.407352    2565 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx" podUID="6a3f6e45-845f-4982-8ee3-6c163fbf4689" containerName="llm-d-routing-sidecar" containerID="cri-o://34323a2e55f0128b4f1b7fa0bb03f8c87be7f6550d3722cccc2c49938c862d3b" gracePeriod=45
Jul 08 17:55:16.611741 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:16.611699    2565 generic.go:356] "Generic (PLEG): container finished" podID="6a3f6e45-845f-4982-8ee3-6c163fbf4689" containerID="df07589598269ba47d1d1ce8491438bd5eff34fabc4f8224bc07efede219e7c2" exitCode=0
Jul 08 17:55:16.611741 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:16.611738    2565 generic.go:356] "Generic (PLEG): container finished" podID="6a3f6e45-845f-4982-8ee3-6c163fbf4689" containerID="34323a2e55f0128b4f1b7fa0bb03f8c87be7f6550d3722cccc2c49938c862d3b" exitCode=0
Jul 08 17:55:16.612010 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:16.611794    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx" event={"ID":"6a3f6e45-845f-4982-8ee3-6c163fbf4689","Type":"ContainerDied","Data":"df07589598269ba47d1d1ce8491438bd5eff34fabc4f8224bc07efede219e7c2"}
Jul 08 17:55:16.612010 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:16.611873    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx" event={"ID":"6a3f6e45-845f-4982-8ee3-6c163fbf4689","Type":"ContainerDied","Data":"34323a2e55f0128b4f1b7fa0bb03f8c87be7f6550d3722cccc2c49938c862d3b"}
Jul 08 17:55:16.671497 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:16.671466    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"
Jul 08 17:55:16.771811 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:16.771767    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmvfr\" (UniqueName: \"kubernetes.io/projected/6a3f6e45-845f-4982-8ee3-6c163fbf4689-kube-api-access-jmvfr\") pod \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\" (UID: \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\") "
Jul 08 17:55:16.771811 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:16.771820    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a3f6e45-845f-4982-8ee3-6c163fbf4689-tls-certs\") pod \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\" (UID: \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\") "
Jul 08 17:55:16.772080 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:16.771883    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a3f6e45-845f-4982-8ee3-6c163fbf4689-model-cache\") pod \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\" (UID: \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\") "
Jul 08 17:55:16.772080 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:16.771905    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a3f6e45-845f-4982-8ee3-6c163fbf4689-dshm\") pod \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\" (UID: \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\") "
Jul 08 17:55:16.772080 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:16.771932    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a3f6e45-845f-4982-8ee3-6c163fbf4689-home\") pod \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\" (UID: \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\") "
Jul 08 17:55:16.772080 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:16.771959    2565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a3f6e45-845f-4982-8ee3-6c163fbf4689-tmp-dir\") pod \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\" (UID: \"6a3f6e45-845f-4982-8ee3-6c163fbf4689\") "
Jul 08 17:55:16.772293 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:16.772196    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3f6e45-845f-4982-8ee3-6c163fbf4689-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a3f6e45-845f-4982-8ee3-6c163fbf4689" (UID: "6a3f6e45-845f-4982-8ee3-6c163fbf4689"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:55:16.772293 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:16.772234    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3f6e45-845f-4982-8ee3-6c163fbf4689-home" (OuterVolumeSpecName: "home") pod "6a3f6e45-845f-4982-8ee3-6c163fbf4689" (UID: "6a3f6e45-845f-4982-8ee3-6c163fbf4689"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:55:16.772293 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:16.772258    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3f6e45-845f-4982-8ee3-6c163fbf4689-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6a3f6e45-845f-4982-8ee3-6c163fbf4689" (UID: "6a3f6e45-845f-4982-8ee3-6c163fbf4689"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:55:16.774107 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:16.774076    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3f6e45-845f-4982-8ee3-6c163fbf4689-dshm" (OuterVolumeSpecName: "dshm") pod "6a3f6e45-845f-4982-8ee3-6c163fbf4689" (UID: "6a3f6e45-845f-4982-8ee3-6c163fbf4689"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 08 17:55:16.774107 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:16.774086    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a3f6e45-845f-4982-8ee3-6c163fbf4689-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a3f6e45-845f-4982-8ee3-6c163fbf4689" (UID: "6a3f6e45-845f-4982-8ee3-6c163fbf4689"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 08 17:55:16.774253 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:16.774160    2565 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a3f6e45-845f-4982-8ee3-6c163fbf4689-kube-api-access-jmvfr" (OuterVolumeSpecName: "kube-api-access-jmvfr") pod "6a3f6e45-845f-4982-8ee3-6c163fbf4689" (UID: "6a3f6e45-845f-4982-8ee3-6c163fbf4689"). InnerVolumeSpecName "kube-api-access-jmvfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 08 17:55:16.873735 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:16.873620    2565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a3f6e45-845f-4982-8ee3-6c163fbf4689-model-cache\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:55:16.873735 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:16.873652    2565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a3f6e45-845f-4982-8ee3-6c163fbf4689-dshm\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:55:16.873735 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:16.873661    2565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a3f6e45-845f-4982-8ee3-6c163fbf4689-home\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:55:16.873735 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:16.873669    2565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a3f6e45-845f-4982-8ee3-6c163fbf4689-tmp-dir\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:55:16.873735 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:16.873710    2565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmvfr\" (UniqueName: \"kubernetes.io/projected/6a3f6e45-845f-4982-8ee3-6c163fbf4689-kube-api-access-jmvfr\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:55:16.873735 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:16.873724    2565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a3f6e45-845f-4982-8ee3-6c163fbf4689-tls-certs\") on node \"ip-10-0-128-142.ec2.internal\" DevicePath \"\""
Jul 08 17:55:17.619559 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:17.619529    2565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"
Jul 08 17:55:17.620070 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:17.619583    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx" event={"ID":"6a3f6e45-845f-4982-8ee3-6c163fbf4689","Type":"ContainerDied","Data":"372d09e84980058b7f248b8a867bd2b84385bd9e5c5ff9bbf0d410464966b3ff"}
Jul 08 17:55:17.620070 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:17.619650    2565 scope.go:117] "RemoveContainer" containerID="df07589598269ba47d1d1ce8491438bd5eff34fabc4f8224bc07efede219e7c2"
Jul 08 17:55:17.629827 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:17.629801    2565 scope.go:117] "RemoveContainer" containerID="34323a2e55f0128b4f1b7fa0bb03f8c87be7f6550d3722cccc2c49938c862d3b"
Jul 08 17:55:17.645895 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:17.645857    2565 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"]
Jul 08 17:55:17.648553 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:17.648523    2565 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/scheduler-v06-threshold-migration-test-kserve-5994569d5b-cn6jx"]
Jul 08 17:55:17.822181 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:17.822141    2565 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a3f6e45-845f-4982-8ee3-6c163fbf4689" path="/var/lib/kubelet/pods/6a3f6e45-845f-4982-8ee3-6c163fbf4689/volumes"
Jul 08 17:55:22.774551 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:22.774512    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:55:22.858601 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:22.858559    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:55:38.767923 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:38.767867    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/0.log"
Jul 08 17:55:38.771976 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:38.771947    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/1.log"
Jul 08 17:55:38.786892 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:38.786855    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/1.log"
Jul 08 17:55:38.787104 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:38.787090    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/0.log"
Jul 08 17:55:38.925321 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:38.925288    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74_c6e3eb6e-ca4d-41fd-a402-927efffc9858/main/0.log"
Jul 08 17:55:40.462422 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:40.462383    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/0.log"
Jul 08 17:55:40.466363 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:40.466336    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/1.log"
Jul 08 17:55:40.485317 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:40.485278    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/1.log"
Jul 08 17:55:40.485503 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:40.485473    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/0.log"
Jul 08 17:55:40.603213 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:40.603177    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74_c6e3eb6e-ca4d-41fd-a402-927efffc9858/main/0.log"
Jul 08 17:55:42.091056 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:42.091021    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/0.log"
Jul 08 17:55:42.095056 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:42.095021    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/1.log"
Jul 08 17:55:42.111144 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:42.111111    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/1.log"
Jul 08 17:55:42.111432 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:42.111346    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/0.log"
Jul 08 17:55:42.219561 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:42.219533    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74_c6e3eb6e-ca4d-41fd-a402-927efffc9858/main/0.log"
Jul 08 17:55:43.721866 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:43.721836    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/0.log"
Jul 08 17:55:43.726266 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:43.726238    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/1.log"
Jul 08 17:55:43.740950 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:43.740913    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/1.log"
Jul 08 17:55:43.741346 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:43.741316    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/0.log"
Jul 08 17:55:43.852303 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:43.852268    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74_c6e3eb6e-ca4d-41fd-a402-927efffc9858/main/0.log"
Jul 08 17:55:45.326836 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:45.326802    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/0.log"
Jul 08 17:55:45.330739 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:45.330711    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/1.log"
Jul 08 17:55:45.346668 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:45.346639    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/1.log"
Jul 08 17:55:45.347050 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:45.347028    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/0.log"
Jul 08 17:55:45.473734 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:45.473697    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74_c6e3eb6e-ca4d-41fd-a402-927efffc9858/main/0.log"
Jul 08 17:55:46.951374 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:46.951337    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/0.log"
Jul 08 17:55:46.955773 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:46.955736    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/1.log"
Jul 08 17:55:46.973989 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:46.973939    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/1.log"
Jul 08 17:55:46.974284 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:46.974258    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/0.log"
Jul 08 17:55:47.122329 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:47.122300    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74_c6e3eb6e-ca4d-41fd-a402-927efffc9858/main/0.log"
Jul 08 17:55:48.774703 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:48.774638    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/0.log"
Jul 08 17:55:48.778740 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:48.778697    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/1.log"
Jul 08 17:55:48.797442 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:48.797408    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/1.log"
Jul 08 17:55:48.797810 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:48.797788    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/0.log"
Jul 08 17:55:48.911164 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:48.911126    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74_c6e3eb6e-ca4d-41fd-a402-927efffc9858/main/0.log"
Jul 08 17:55:50.527542 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:50.527508    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/0.log"
Jul 08 17:55:50.531827 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:50.531796    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/1.log"
Jul 08 17:55:50.549495 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:50.549461    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/1.log"
Jul 08 17:55:50.549785 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:50.549760    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/0.log"
Jul 08 17:55:50.668525 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:50.668490    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74_c6e3eb6e-ca4d-41fd-a402-927efffc9858/main/0.log"
Jul 08 17:55:52.232925 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:52.232889    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/0.log"
Jul 08 17:55:52.236474 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:52.236448    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/1.log"
Jul 08 17:55:52.256481 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:52.256434    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/1.log"
Jul 08 17:55:52.256806 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:52.256788    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/0.log"
Jul 08 17:55:52.382636 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:52.382598    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74_c6e3eb6e-ca4d-41fd-a402-927efffc9858/main/0.log"
Jul 08 17:55:53.908374 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:53.908336    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/0.log"
Jul 08 17:55:53.912568 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:53.912539    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/1.log"
Jul 08 17:55:53.927768 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:53.927737    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/1.log"
Jul 08 17:55:53.928032 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:53.927888    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/0.log"
Jul 08 17:55:54.040233 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:54.040200    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74_c6e3eb6e-ca4d-41fd-a402-927efffc9858/main/0.log"
Jul 08 17:55:55.540194 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:55.540148    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/0.log"
Jul 08 17:55:55.544303 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:55.544276    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/1.log"
Jul 08 17:55:55.567557 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:55.567528    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/1.log"
Jul 08 17:55:55.567985 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:55.567960    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/0.log"
Jul 08 17:55:55.684954 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:55.684922    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74_c6e3eb6e-ca4d-41fd-a402-927efffc9858/main/0.log"
Jul 08 17:55:57.160098 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:57.160061    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/0.log"
Jul 08 17:55:57.164374 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:57.164331    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/1.log"
Jul 08 17:55:57.179979 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:57.179940    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/1.log"
Jul 08 17:55:57.180281 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:57.180259    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/0.log"
Jul 08 17:55:57.293198 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:57.293157    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74_c6e3eb6e-ca4d-41fd-a402-927efffc9858/main/0.log"
Jul 08 17:55:58.758692 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:58.758632    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/0.log"
Jul 08 17:55:58.763787 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:58.763753    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/1.log"
Jul 08 17:55:58.779315 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:58.779280    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/1.log"
Jul 08 17:55:58.779768 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:58.779746    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/0.log"
Jul 08 17:55:58.887324 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:55:58.887290    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74_c6e3eb6e-ca4d-41fd-a402-927efffc9858/main/0.log"
Jul 08 17:56:00.372853 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:00.372824    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/0.log"
Jul 08 17:56:00.376952 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:00.376927    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/1.log"
Jul 08 17:56:00.394384 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:00.394358    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/1.log"
Jul 08 17:56:00.394745 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:00.394722    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/0.log"
Jul 08 17:56:00.503035 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:00.503002    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74_c6e3eb6e-ca4d-41fd-a402-927efffc9858/main/0.log"
Jul 08 17:56:02.001434 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:02.001402    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/0.log"
Jul 08 17:56:02.005584 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:02.005555    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/1.log"
Jul 08 17:56:02.020974 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:02.020942    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/1.log"
Jul 08 17:56:02.021263 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:02.021243    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/0.log"
Jul 08 17:56:02.134537 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:02.134505    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74_c6e3eb6e-ca4d-41fd-a402-927efffc9858/main/0.log"
Jul 08 17:56:03.606291 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:03.606262    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/0.log"
Jul 08 17:56:03.610725 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:03.610702    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-1-openshift-default-54c47cdfc7-rxfpz_279f6cb5-9686-4583-b06c-bd27b344f0a9/istio-proxy/1.log"
Jul 08 17:56:03.625441 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:03.625395    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/1.log"
Jul 08 17:56:03.625557 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:03.625508    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_router-gateway-2-openshift-default-684dc9b4b9-h2x2d_49d08e4c-b706-4071-a94c-26aca5a1fa9a/istio-proxy/0.log"
Jul 08 17:56:03.768510 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:03.768484    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_scheduler-v06-pd-migration-test-kserve-router-scheduler-595ls74_c6e3eb6e-ca4d-41fd-a402-927efffc9858/main/0.log"
Jul 08 17:56:05.629293 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:05.629262    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-5kc7r_63601443-3f06-49ac-ada3-05f45b6924d2/discovery/0.log"
Jul 08 17:56:07.067398 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:07.067367    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-5kc7r_63601443-3f06-49ac-ada3-05f45b6924d2/discovery/0.log"
Jul 08 17:56:08.497963 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:08.497928    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-5kc7r_63601443-3f06-49ac-ada3-05f45b6924d2/discovery/0.log"
Jul 08 17:56:10.054815 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:10.054779    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-5kc7r_63601443-3f06-49ac-ada3-05f45b6924d2/discovery/0.log"
Jul 08 17:56:11.228193 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:11.228162    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-lkzck_b74fcdd0-709a-4fc2-bd48-8a6b345b1f92/authorino/0.log"
Jul 08 17:56:11.254829 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:11.254804    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-hp27q_a74002bb-6e21-4774-88a0-60f3e557ac54/manager/0.log"
Jul 08 17:56:11.270375 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:11.270353    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-dx5b4_525ed767-7832-403b-899f-1443f130dadb/manager/0.log"
Jul 08 17:56:11.458956 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:11.458929    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-tldlw_e7a29723-2210-4ce6-acfe-711776cdb84c/manager/0.log"
Jul 08 17:56:12.666574 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:12.666548    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-lkzck_b74fcdd0-709a-4fc2-bd48-8a6b345b1f92/authorino/0.log"
Jul 08 17:56:12.687812 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:12.687787    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-hp27q_a74002bb-6e21-4774-88a0-60f3e557ac54/manager/0.log"
Jul 08 17:56:12.720505 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:12.720481    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-dx5b4_525ed767-7832-403b-899f-1443f130dadb/manager/0.log"
Jul 08 17:56:12.929335 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:12.929306    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-tldlw_e7a29723-2210-4ce6-acfe-711776cdb84c/manager/0.log"
Jul 08 17:56:14.082440 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:14.082410    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-lkzck_b74fcdd0-709a-4fc2-bd48-8a6b345b1f92/authorino/0.log"
Jul 08 17:56:14.101316 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:14.101284    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-hp27q_a74002bb-6e21-4774-88a0-60f3e557ac54/manager/0.log"
Jul 08 17:56:14.118213 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:14.118176    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-dx5b4_525ed767-7832-403b-899f-1443f130dadb/manager/0.log"
Jul 08 17:56:14.310179 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:14.310155    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-tldlw_e7a29723-2210-4ce6-acfe-711776cdb84c/manager/0.log"
Jul 08 17:56:15.599538 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:15.599501    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-lkzck_b74fcdd0-709a-4fc2-bd48-8a6b345b1f92/authorino/0.log"
Jul 08 17:56:15.654226 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:15.654193    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-hp27q_a74002bb-6e21-4774-88a0-60f3e557ac54/manager/0.log"
Jul 08 17:56:15.708404 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:15.708376    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-dx5b4_525ed767-7832-403b-899f-1443f130dadb/manager/0.log"
Jul 08 17:56:15.889024 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:15.888948    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-tldlw_e7a29723-2210-4ce6-acfe-711776cdb84c/manager/0.log"
Jul 08 17:56:17.137867 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:17.137841    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-lkzck_b74fcdd0-709a-4fc2-bd48-8a6b345b1f92/authorino/0.log"
Jul 08 17:56:17.298053 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:17.298027    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-hp27q_a74002bb-6e21-4774-88a0-60f3e557ac54/manager/0.log"
Jul 08 17:56:17.317083 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:17.317053    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-dx5b4_525ed767-7832-403b-899f-1443f130dadb/manager/0.log"
Jul 08 17:56:17.541645 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:17.541620    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-tldlw_e7a29723-2210-4ce6-acfe-711776cdb84c/manager/0.log"
Jul 08 17:56:18.738547 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:18.738514    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-lkzck_b74fcdd0-709a-4fc2-bd48-8a6b345b1f92/authorino/0.log"
Jul 08 17:56:18.756293 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:18.756267    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-hp27q_a74002bb-6e21-4774-88a0-60f3e557ac54/manager/0.log"
Jul 08 17:56:18.775030 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:18.775002    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-dx5b4_525ed767-7832-403b-899f-1443f130dadb/manager/0.log"
Jul 08 17:56:18.971143 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:18.971117    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-tldlw_e7a29723-2210-4ce6-acfe-711776cdb84c/manager/0.log"
Jul 08 17:56:20.131520 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:20.131491    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-lkzck_b74fcdd0-709a-4fc2-bd48-8a6b345b1f92/authorino/0.log"
Jul 08 17:56:20.148066 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:20.148038    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-hp27q_a74002bb-6e21-4774-88a0-60f3e557ac54/manager/0.log"
Jul 08 17:56:20.162572 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:20.162552    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-dx5b4_525ed767-7832-403b-899f-1443f130dadb/manager/0.log"
Jul 08 17:56:20.363158 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:20.363125    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-tldlw_e7a29723-2210-4ce6-acfe-711776cdb84c/manager/0.log"
Jul 08 17:56:21.501790 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:21.501752    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-lkzck_b74fcdd0-709a-4fc2-bd48-8a6b345b1f92/authorino/0.log"
Jul 08 17:56:21.517737 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:21.517713    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-hp27q_a74002bb-6e21-4774-88a0-60f3e557ac54/manager/0.log"
Jul 08 17:56:21.531062 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:21.531031    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-dx5b4_525ed767-7832-403b-899f-1443f130dadb/manager/0.log"
Jul 08 17:56:21.726466 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:21.726430    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-tldlw_e7a29723-2210-4ce6-acfe-711776cdb84c/manager/0.log"
Jul 08 17:56:22.891084 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:22.891052    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-lkzck_b74fcdd0-709a-4fc2-bd48-8a6b345b1f92/authorino/0.log"
Jul 08 17:56:22.906541 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:22.906521    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-hp27q_a74002bb-6e21-4774-88a0-60f3e557ac54/manager/0.log"
Jul 08 17:56:22.921270 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:22.921246    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-dx5b4_525ed767-7832-403b-899f-1443f130dadb/manager/0.log"
Jul 08 17:56:23.123347 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:23.123315    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-tldlw_e7a29723-2210-4ce6-acfe-711776cdb84c/manager/0.log"
Jul 08 17:56:24.247818 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:24.247786    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-lkzck_b74fcdd0-709a-4fc2-bd48-8a6b345b1f92/authorino/0.log"
Jul 08 17:56:24.265623 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:24.265593    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-hp27q_a74002bb-6e21-4774-88a0-60f3e557ac54/manager/0.log"
Jul 08 17:56:24.279902 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:24.279867    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-dx5b4_525ed767-7832-403b-899f-1443f130dadb/manager/0.log"
Jul 08 17:56:24.470110 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:24.470079    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-tldlw_e7a29723-2210-4ce6-acfe-711776cdb84c/manager/0.log"
Jul 08 17:56:25.595098 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:25.595064    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-lkzck_b74fcdd0-709a-4fc2-bd48-8a6b345b1f92/authorino/0.log"
Jul 08 17:56:25.610614 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:25.610591    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-hp27q_a74002bb-6e21-4774-88a0-60f3e557ac54/manager/0.log"
Jul 08 17:56:25.625689 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:25.625659    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-dx5b4_525ed767-7832-403b-899f-1443f130dadb/manager/0.log"
Jul 08 17:56:25.821420 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:25.821392    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-tldlw_e7a29723-2210-4ce6-acfe-711776cdb84c/manager/0.log"
Jul 08 17:56:28.340202 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:28.340161    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:56:28.473111 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:28.473075    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wf794/must-gather-4m7rl"]
Jul 08 17:56:28.473481 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:28.473467    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a3f6e45-845f-4982-8ee3-6c163fbf4689" containerName="llm-d-routing-sidecar"
Jul 08 17:56:28.473527 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:28.473483    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a3f6e45-845f-4982-8ee3-6c163fbf4689" containerName="llm-d-routing-sidecar"
Jul 08 17:56:28.473527 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:28.473491    2565 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a3f6e45-845f-4982-8ee3-6c163fbf4689" containerName="main"
Jul 08 17:56:28.473527 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:28.473497    2565 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a3f6e45-845f-4982-8ee3-6c163fbf4689" containerName="main"
Jul 08 17:56:28.473628 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:28.473557    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a3f6e45-845f-4982-8ee3-6c163fbf4689" containerName="main"
Jul 08 17:56:28.473628 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:28.473567    2565 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a3f6e45-845f-4982-8ee3-6c163fbf4689" containerName="llm-d-routing-sidecar"
Jul 08 17:56:28.476866 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:28.476846    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wf794/must-gather-4m7rl"
Jul 08 17:56:28.479356 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:28.479328    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wf794\"/\"kube-root-ca.crt\""
Jul 08 17:56:28.479475 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:28.479399    2565 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wf794\"/\"default-dockercfg-97555\""
Jul 08 17:56:28.479475 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:28.479402    2565 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wf794\"/\"openshift-service-ca.crt\""
Jul 08 17:56:28.486002 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:28.485927    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:56:28.487345 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:28.487324    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wf794/must-gather-4m7rl"]
Jul 08 17:56:28.588774 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:28.588746    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:56:28.590784 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:28.590726    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4a7ada88-06bf-402f-8e65-510f3c9903fd-must-gather-output\") pod \"must-gather-4m7rl\" (UID: \"4a7ada88-06bf-402f-8e65-510f3c9903fd\") " pod="openshift-must-gather-wf794/must-gather-4m7rl"
Jul 08 17:56:28.590872 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:28.590836    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jw4g\" (UniqueName: \"kubernetes.io/projected/4a7ada88-06bf-402f-8e65-510f3c9903fd-kube-api-access-9jw4g\") pod \"must-gather-4m7rl\" (UID: \"4a7ada88-06bf-402f-8e65-510f3c9903fd\") " pod="openshift-must-gather-wf794/must-gather-4m7rl"
Jul 08 17:56:28.691988 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:28.691960    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jw4g\" (UniqueName: \"kubernetes.io/projected/4a7ada88-06bf-402f-8e65-510f3c9903fd-kube-api-access-9jw4g\") pod \"must-gather-4m7rl\" (UID: \"4a7ada88-06bf-402f-8e65-510f3c9903fd\") " pod="openshift-must-gather-wf794/must-gather-4m7rl"
Jul 08 17:56:28.692114 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:28.692016    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4a7ada88-06bf-402f-8e65-510f3c9903fd-must-gather-output\") pod \"must-gather-4m7rl\" (UID: \"4a7ada88-06bf-402f-8e65-510f3c9903fd\") " pod="openshift-must-gather-wf794/must-gather-4m7rl"
Jul 08 17:56:28.692340 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:28.692324    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4a7ada88-06bf-402f-8e65-510f3c9903fd-must-gather-output\") pod \"must-gather-4m7rl\" (UID: \"4a7ada88-06bf-402f-8e65-510f3c9903fd\") " pod="openshift-must-gather-wf794/must-gather-4m7rl"
Jul 08 17:56:28.701997 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:28.701969    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jw4g\" (UniqueName: \"kubernetes.io/projected/4a7ada88-06bf-402f-8e65-510f3c9903fd-kube-api-access-9jw4g\") pod \"must-gather-4m7rl\" (UID: \"4a7ada88-06bf-402f-8e65-510f3c9903fd\") " pod="openshift-must-gather-wf794/must-gather-4m7rl"
Jul 08 17:56:28.787296 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:28.787271    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wf794/must-gather-4m7rl"
Jul 08 17:56:28.918494 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:28.918464    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wf794/must-gather-4m7rl"]
Jul 08 17:56:28.919758 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:56:28.919731    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a7ada88_06bf_402f_8e65_510f3c9903fd.slice/crio-f0d34a5e0a184d314dedee3c2bc2cc929967717e101930fde1168ef9a7b129c7 WatchSource:0}: Error finding container f0d34a5e0a184d314dedee3c2bc2cc929967717e101930fde1168ef9a7b129c7: Status 404 returned error can't find the container with id f0d34a5e0a184d314dedee3c2bc2cc929967717e101930fde1168ef9a7b129c7
Jul 08 17:56:28.921618 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:28.921601    2565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 08 17:56:29.006736 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:29.006709    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wf794/must-gather-4m7rl" event={"ID":"4a7ada88-06bf-402f-8e65-510f3c9903fd","Type":"ContainerStarted","Data":"f0d34a5e0a184d314dedee3c2bc2cc929967717e101930fde1168ef9a7b129c7"}
Jul 08 17:56:30.015800 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:30.015713    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wf794/must-gather-4m7rl" event={"ID":"4a7ada88-06bf-402f-8e65-510f3c9903fd","Type":"ContainerStarted","Data":"5365bd850d4ecee47aee186ebeb4cefbeffc1c477d9868034af08c69b3b0e2eb"}
Jul 08 17:56:30.015800 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:30.015760    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wf794/must-gather-4m7rl" event={"ID":"4a7ada88-06bf-402f-8e65-510f3c9903fd","Type":"ContainerStarted","Data":"9e27d209f7172c29233154f36ed8960a13942a20fe57f708f1ddea16401db87b"}
Jul 08 17:56:30.036338 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:30.036289    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wf794/must-gather-4m7rl" podStartSLOduration=1.228890043 podStartE2EDuration="2.036273391s" totalImagesPullingTime="807.383348ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 17:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:56:30.034302533 +0000 UTC m=+4926.747663961" watchObservedRunningTime="2026-07-08 17:56:30.036273391 +0000 UTC m=+4926.749634817"
Jul 08 17:56:31.435001 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:31.434968    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-sxzrc_3cf84d3d-0028-4d7f-b178-c4a0d8391cdc/global-pull-secret-syncer/0.log"
Jul 08 17:56:31.542865 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:31.542836    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log"
Jul 08 17:56:31.663228 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:31.663192    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-142.ec2.internal_6c8c7615c18634239b1c0970c2d029c0/haproxy/0.log"
Jul 08 17:56:35.148156 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:35.148119    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:56:35.248224 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:35.248189    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:56:35.417744 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:35.417111    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7c77957575-lkzck_b74fcdd0-709a-4fc2-bd48-8a6b345b1f92/authorino/0.log"
Jul 08 17:56:35.460357 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:35.460288    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-6d85f6564-hp27q_a74002bb-6e21-4774-88a0-60f3e557ac54/manager/0.log"
Jul 08 17:56:35.504651 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:35.504621    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-65b49595d7-dx5b4_525ed767-7832-403b-899f-1443f130dadb/manager/0.log"
Jul 08 17:56:35.788212 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:35.788056    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-bb7ddb7d7-tldlw_e7a29723-2210-4ce6-acfe-711776cdb84c/manager/0.log"
Jul 08 17:56:37.575279 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:37.575240    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_35d5abaa-a8a0-46be-bc8e-7b789df4dfc6/alertmanager/0.log"
Jul 08 17:56:37.606696 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:37.606646    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_35d5abaa-a8a0-46be-bc8e-7b789df4dfc6/config-reloader/0.log"
Jul 08 17:56:37.632550 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:37.632511    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_35d5abaa-a8a0-46be-bc8e-7b789df4dfc6/kube-rbac-proxy-web/0.log"
Jul 08 17:56:37.657964 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:37.657895    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_35d5abaa-a8a0-46be-bc8e-7b789df4dfc6/kube-rbac-proxy/0.log"
Jul 08 17:56:37.689935 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:37.689908    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_35d5abaa-a8a0-46be-bc8e-7b789df4dfc6/kube-rbac-proxy-metric/0.log"
Jul 08 17:56:37.715123 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:37.715099    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_35d5abaa-a8a0-46be-bc8e-7b789df4dfc6/prom-label-proxy/0.log"
Jul 08 17:56:37.749839 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:37.749807    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_35d5abaa-a8a0-46be-bc8e-7b789df4dfc6/init-config-reloader/0.log"
Jul 08 17:56:37.838231 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:37.838145    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-b8mjr_53b2bf60-5387-46eb-b264-e0410b55f8b9/kube-state-metrics/0.log"
Jul 08 17:56:37.861211 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:37.861184    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-b8mjr_53b2bf60-5387-46eb-b264-e0410b55f8b9/kube-rbac-proxy-main/0.log"
Jul 08 17:56:37.888914 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:37.888879    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-b8mjr_53b2bf60-5387-46eb-b264-e0410b55f8b9/kube-rbac-proxy-self/0.log"
Jul 08 17:56:37.956025 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:37.955997    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-t9vpw_b6393836-702c-4035-9e28-463bf38007f6/monitoring-plugin/0.log"
Jul 08 17:56:38.182219 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:38.182189    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zsdvl_3898e889-6916-443f-9d83-dccd019d6544/node-exporter/0.log"
Jul 08 17:56:38.212270 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:38.212242    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zsdvl_3898e889-6916-443f-9d83-dccd019d6544/kube-rbac-proxy/0.log"
Jul 08 17:56:38.250980 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:38.250954    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zsdvl_3898e889-6916-443f-9d83-dccd019d6544/init-textfile/0.log"
Jul 08 17:56:38.835331 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:38.835302    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-646c8d68d4-kd7vt_a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee/telemeter-client/0.log"
Jul 08 17:56:38.896221 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:38.896192    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-646c8d68d4-kd7vt_a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee/reload/0.log"
Jul 08 17:56:38.941571 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:38.941541    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-646c8d68d4-kd7vt_a1bc2786-a23b-470f-bf3d-2a9cd51ff2ee/kube-rbac-proxy/0.log"
Jul 08 17:56:41.476170 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:41.476141    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-85d9d9b968-h2892_9a2587bc-4630-45e3-b984-5c917d0301e7/console/0.log"
Jul 08 17:56:42.823291 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:42.823252    2565 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wf794/perf-node-gather-daemonset-drcfb"]
Jul 08 17:56:42.829311 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:42.829284    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wf794/perf-node-gather-daemonset-drcfb"
Jul 08 17:56:42.839964 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:42.839218    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wf794/perf-node-gather-daemonset-drcfb"]
Jul 08 17:56:42.845316 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:42.845283    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1aefd1d7-3484-4d9b-a706-1f1c30d91f20-sys\") pod \"perf-node-gather-daemonset-drcfb\" (UID: \"1aefd1d7-3484-4d9b-a706-1f1c30d91f20\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-drcfb"
Jul 08 17:56:42.845450 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:42.845328    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1aefd1d7-3484-4d9b-a706-1f1c30d91f20-proc\") pod \"perf-node-gather-daemonset-drcfb\" (UID: \"1aefd1d7-3484-4d9b-a706-1f1c30d91f20\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-drcfb"
Jul 08 17:56:42.845450 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:42.845416    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1aefd1d7-3484-4d9b-a706-1f1c30d91f20-podres\") pod \"perf-node-gather-daemonset-drcfb\" (UID: \"1aefd1d7-3484-4d9b-a706-1f1c30d91f20\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-drcfb"
Jul 08 17:56:42.845567 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:42.845452    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sksw\" (UniqueName: \"kubernetes.io/projected/1aefd1d7-3484-4d9b-a706-1f1c30d91f20-kube-api-access-9sksw\") pod \"perf-node-gather-daemonset-drcfb\" (UID: \"1aefd1d7-3484-4d9b-a706-1f1c30d91f20\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-drcfb"
Jul 08 17:56:42.845567 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:42.845476    2565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1aefd1d7-3484-4d9b-a706-1f1c30d91f20-lib-modules\") pod \"perf-node-gather-daemonset-drcfb\" (UID: \"1aefd1d7-3484-4d9b-a706-1f1c30d91f20\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-drcfb"
Jul 08 17:56:42.946237 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:42.946185    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1aefd1d7-3484-4d9b-a706-1f1c30d91f20-podres\") pod \"perf-node-gather-daemonset-drcfb\" (UID: \"1aefd1d7-3484-4d9b-a706-1f1c30d91f20\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-drcfb"
Jul 08 17:56:42.946471 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:42.946243    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sksw\" (UniqueName: \"kubernetes.io/projected/1aefd1d7-3484-4d9b-a706-1f1c30d91f20-kube-api-access-9sksw\") pod \"perf-node-gather-daemonset-drcfb\" (UID: \"1aefd1d7-3484-4d9b-a706-1f1c30d91f20\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-drcfb"
Jul 08 17:56:42.946471 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:42.946268    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1aefd1d7-3484-4d9b-a706-1f1c30d91f20-lib-modules\") pod \"perf-node-gather-daemonset-drcfb\" (UID: \"1aefd1d7-3484-4d9b-a706-1f1c30d91f20\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-drcfb"
Jul 08 17:56:42.946471 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:42.946299    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1aefd1d7-3484-4d9b-a706-1f1c30d91f20-sys\") pod \"perf-node-gather-daemonset-drcfb\" (UID: \"1aefd1d7-3484-4d9b-a706-1f1c30d91f20\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-drcfb"
Jul 08 17:56:42.946471 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:42.946340    2565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1aefd1d7-3484-4d9b-a706-1f1c30d91f20-proc\") pod \"perf-node-gather-daemonset-drcfb\" (UID: \"1aefd1d7-3484-4d9b-a706-1f1c30d91f20\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-drcfb"
Jul 08 17:56:42.946471 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:42.946399    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1aefd1d7-3484-4d9b-a706-1f1c30d91f20-podres\") pod \"perf-node-gather-daemonset-drcfb\" (UID: \"1aefd1d7-3484-4d9b-a706-1f1c30d91f20\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-drcfb"
Jul 08 17:56:42.946471 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:42.946432    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1aefd1d7-3484-4d9b-a706-1f1c30d91f20-sys\") pod \"perf-node-gather-daemonset-drcfb\" (UID: \"1aefd1d7-3484-4d9b-a706-1f1c30d91f20\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-drcfb"
Jul 08 17:56:42.946471 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:42.946443    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1aefd1d7-3484-4d9b-a706-1f1c30d91f20-proc\") pod \"perf-node-gather-daemonset-drcfb\" (UID: \"1aefd1d7-3484-4d9b-a706-1f1c30d91f20\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-drcfb"
Jul 08 17:56:42.946471 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:42.946452    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1aefd1d7-3484-4d9b-a706-1f1c30d91f20-lib-modules\") pod \"perf-node-gather-daemonset-drcfb\" (UID: \"1aefd1d7-3484-4d9b-a706-1f1c30d91f20\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-drcfb"
Jul 08 17:56:42.956148 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:42.956128    2565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sksw\" (UniqueName: \"kubernetes.io/projected/1aefd1d7-3484-4d9b-a706-1f1c30d91f20-kube-api-access-9sksw\") pod \"perf-node-gather-daemonset-drcfb\" (UID: \"1aefd1d7-3484-4d9b-a706-1f1c30d91f20\") " pod="openshift-must-gather-wf794/perf-node-gather-daemonset-drcfb"
Jul 08 17:56:43.144634 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:43.144555    2565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wf794/perf-node-gather-daemonset-drcfb"
Jul 08 17:56:43.301877 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:43.301842    2565 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wf794/perf-node-gather-daemonset-drcfb"]
Jul 08 17:56:43.302870 ip-10-0-128-142 kubenswrapper[2565]: W0708 17:56:43.302838    2565 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod1aefd1d7_3484_4d9b_a706_1f1c30d91f20.slice/crio-43b4d619a0b236c63ac0d70fdb7b76ed5555ae4e288d766b2e3199026b738eb7 WatchSource:0}: Error finding container 43b4d619a0b236c63ac0d70fdb7b76ed5555ae4e288d766b2e3199026b738eb7: Status 404 returned error can't find the container with id 43b4d619a0b236c63ac0d70fdb7b76ed5555ae4e288d766b2e3199026b738eb7
Jul 08 17:56:43.327024 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:43.326926    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tf2qd_187d83d2-dbc5-46aa-8e7b-f3df420b474c/dns/0.log"
Jul 08 17:56:43.351290 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:43.351265    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tf2qd_187d83d2-dbc5-46aa-8e7b-f3df420b474c/kube-rbac-proxy/0.log"
Jul 08 17:56:43.402869 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:43.402796    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hq9qt_1d51073c-aced-4933-a4f7-0363e8020735/dns-node-resolver/0.log"
Jul 08 17:56:44.047793 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:44.047746    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lscjn_6f378482-a771-45ff-85cb-3d9b4b887806/node-ca/0.log"
Jul 08 17:56:44.139494 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:44.139449    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wf794/perf-node-gather-daemonset-drcfb" event={"ID":"1aefd1d7-3484-4d9b-a706-1f1c30d91f20","Type":"ContainerStarted","Data":"364de502f5a24ac4267812d06d35525de889bfd920da04ea217d79f4362a4116"}
Jul 08 17:56:44.139700 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:44.139517    2565 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wf794/perf-node-gather-daemonset-drcfb" event={"ID":"1aefd1d7-3484-4d9b-a706-1f1c30d91f20","Type":"ContainerStarted","Data":"43b4d619a0b236c63ac0d70fdb7b76ed5555ae4e288d766b2e3199026b738eb7"}
Jul 08 17:56:44.140357 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:44.140332    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wf794/perf-node-gather-daemonset-drcfb"
Jul 08 17:56:44.161417 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:44.161365    2565 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wf794/perf-node-gather-daemonset-drcfb" podStartSLOduration=2.161345577 podStartE2EDuration="2.161345577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 17:56:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 17:56:44.160000085 +0000 UTC m=+4940.873361513" watchObservedRunningTime="2026-07-08 17:56:44.161345577 +0000 UTC m=+4940.874707005"
Jul 08 17:56:45.555142 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:45.555050    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-5kc7r_63601443-3f06-49ac-ada3-05f45b6924d2/discovery/0.log"
Jul 08 17:56:46.237788 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:46.237757    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qgtgq_d28f25d1-2748-43e3-95a8-eb74a25f8842/serve-healthcheck-canary/0.log"
Jul 08 17:56:46.770242 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:46.770215    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-24ccq_59fa9d8c-d7a1-430d-bd58-560305b06ec9/kube-rbac-proxy/0.log"
Jul 08 17:56:46.796867 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:46.796839    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-24ccq_59fa9d8c-d7a1-430d-bd58-560305b06ec9/exporter/0.log"
Jul 08 17:56:46.824046 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:46.824005    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-24ccq_59fa9d8c-d7a1-430d-bd58-560305b06ec9/extractor/0.log"
Jul 08 17:56:50.333531 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:50.333485    2565 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6qgmf_42d7b641-e70b-4a9d-b7ee-2acab63c9942/konnectivity-agent/0.log" limit=1024
Jul 08 17:56:50.551535 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:50.551502    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-687b7449bc-w4q6g_abab7113-b73a-446b-bff7-f48652ba2e7e/manager/0.log"
Jul 08 17:56:50.616416 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:50.616334    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-qrsqp_a234a1fb-078c-4982-b8c3-cab134ea4577/server/0.log"
Jul 08 17:56:51.166546 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:51.166514    2565 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wf794/perf-node-gather-daemonset-drcfb"
Jul 08 17:56:51.524169 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:51.524132    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-ps8wt_f71f5666-76c2-46a0-a129-ceacedb14a92/manager/0.log"
Jul 08 17:56:52.228734 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:52.228699    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-5kc7r_63601443-3f06-49ac-ada3-05f45b6924d2/discovery/0.log"
Jul 08 17:56:53.812268 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:53.812211    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-5kc7r_63601443-3f06-49ac-ada3-05f45b6924d2/discovery/0.log"
Jul 08 17:56:58.871791 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:58.871759    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-97xhl_8086c91a-48e3-4158-a6ba-004f9046fa09/kube-multus-additional-cni-plugins/0.log"
Jul 08 17:56:58.898311 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:58.898289    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-97xhl_8086c91a-48e3-4158-a6ba-004f9046fa09/egress-router-binary-copy/0.log"
Jul 08 17:56:58.921946 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:58.921925    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-97xhl_8086c91a-48e3-4158-a6ba-004f9046fa09/cni-plugins/0.log"
Jul 08 17:56:58.947457 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:58.947439    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-97xhl_8086c91a-48e3-4158-a6ba-004f9046fa09/bond-cni-plugin/0.log"
Jul 08 17:56:58.972876 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:58.972851    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-97xhl_8086c91a-48e3-4158-a6ba-004f9046fa09/routeoverride-cni/0.log"
Jul 08 17:56:58.997860 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:58.997841    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-97xhl_8086c91a-48e3-4158-a6ba-004f9046fa09/whereabouts-cni-bincopy/0.log"
Jul 08 17:56:59.023426 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:59.023410    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-97xhl_8086c91a-48e3-4158-a6ba-004f9046fa09/whereabouts-cni/0.log"
Jul 08 17:56:59.276941 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:59.276912    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-t8qnb_7b9bc9aa-a1bc-4af2-8dff-21213565dd18/kube-multus/0.log"
Jul 08 17:56:59.303248 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:59.303216    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bxjqc_d14d465c-2bea-496d-8112-450e25ae7717/network-metrics-daemon/0.log"
Jul 08 17:56:59.323335 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:56:59.323311    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-bxjqc_d14d465c-2bea-496d-8112-450e25ae7717/kube-rbac-proxy/0.log"
Jul 08 17:57:00.296108 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:57:00.296081    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb8xd_aea71a23-3932-4b4d-8a12-57ef7d976049/ovn-controller/0.log"
Jul 08 17:57:00.312770 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:57:00.312745    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb8xd_aea71a23-3932-4b4d-8a12-57ef7d976049/ovn-acl-logging/0.log"
Jul 08 17:57:00.339514 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:57:00.339475    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb8xd_aea71a23-3932-4b4d-8a12-57ef7d976049/kube-rbac-proxy-node/0.log"
Jul 08 17:57:00.365171 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:57:00.365141    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb8xd_aea71a23-3932-4b4d-8a12-57ef7d976049/kube-rbac-proxy-ovn-metrics/0.log"
Jul 08 17:57:00.387172 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:57:00.387148    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb8xd_aea71a23-3932-4b4d-8a12-57ef7d976049/northd/0.log"
Jul 08 17:57:00.410136 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:57:00.410115    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb8xd_aea71a23-3932-4b4d-8a12-57ef7d976049/nbdb/0.log"
Jul 08 17:57:00.433646 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:57:00.433623    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb8xd_aea71a23-3932-4b4d-8a12-57ef7d976049/sbdb/0.log"
Jul 08 17:57:00.531073 ip-10-0-128-142 kubenswrapper[2565]: I0708 17:57:00.531045    2565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gb8xd_aea71a23-3932-4b4d-8a12-57ef7d976049/ovnkube-controller/0.log"